{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用LightGBM/XGboost实现Happy Customer Bank目标客户（贷款成功的客户）识别\n",
    "\n",
    "\n",
    "一、\t任务说明：Happy Customer Bank目标客户识别\n",
    "https://discuss.analyticsvidhya.com/t/hackathon-3-x-predict-customer-worth-for-happy-customer-bank/3802\n",
    "\n",
    "1)\t文件说明\n",
    "Train.csv：训练数据\n",
    "Test.csv：测试数据\n",
    "\n",
    "2)\t字段说明\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "1.\tID - 唯一ID（不能用于预测）\n",
    "2.\tGender - 性别\n",
    "3.\tCity - 城市\n",
    "4.\tMonthly_Income - 月收入（以卢比为单位）\n",
    "5.\tDOB - 出生日期\n",
    "6.\tLead_Creation_Date - 潜在（贷款）创建日期\n",
    "7.\tLoan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "8.\tLoan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "9.\tExisting_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "10.\tEmployer_Name雇主名称\n",
    "11.\tSalary_Account - 薪资帐户银行\n",
    "12.\tMobile_Verified - 是否移动验证（Y / N）\n",
    "13.\tVAR5 - 连续型变量\n",
    "14.\tVAR1-  类别型变量\n",
    "15.\tLoan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "16.\tLoan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "17.\tInterest_Rate - 提交贷款金额的利率\n",
    "18.\tProcessing_Fee - 提交贷款的处理费（INR）\n",
    "19.\tEMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "20.\tFilled_Form - 后期报价后是否已填写申请表格\n",
    "21.\tDevice_Type - 进行申请的设备（浏览器/移动设备）\n",
    "22.\tVar2 - 类别型变量\n",
    "23.\tSource - 类别型变量\n",
    "24.\tVar4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "25.\tLoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）\n",
    "\n",
    "二、作业要求：\n",
    "1.\t适当的特征工程（20分）\n",
    "2.\t用LightGBM完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、num_leaves、max_depth、min_data_in_leaf、colsample_bytree、subsample）进行调优。（70分）\n",
    "或者用XGBoost完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、max_depth、min_child_weight、colsample_bytree、subsample、reg_lambda、reg_）进行调优。\n",
    "3.\t对最终模型给出特征重要性（10分）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入必要的工具包\n",
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>City</th>\n",
       "      <th>Monthly_Income</th>\n",
       "      <th>Loan_Amount_Applied</th>\n",
       "      <th>Loan_Tenure_Applied</th>\n",
       "      <th>Existing_EMI</th>\n",
       "      <th>Employer_Name</th>\n",
       "      <th>Salary_Account</th>\n",
       "      <th>Mobile_Verified</th>\n",
       "      <th>Var5</th>\n",
       "      <th>...</th>\n",
       "      <th>Interest_Rate</th>\n",
       "      <th>Processing_Fee</th>\n",
       "      <th>EMI_Loan_Submitted</th>\n",
       "      <th>Filled_Form</th>\n",
       "      <th>Device_Type</th>\n",
       "      <th>Var2</th>\n",
       "      <th>Source</th>\n",
       "      <th>Var4</th>\n",
       "      <th>LoggedIn</th>\n",
       "      <th>Disbursed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Female</td>\n",
       "      <td>Delhi</td>\n",
       "      <td>20000</td>\n",
       "      <td>300000.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>CYBOSOL</td>\n",
       "      <td>HDFC Bank</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>G</td>\n",
       "      <td>S122</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Male</td>\n",
       "      <td>Mumbai</td>\n",
       "      <td>35000</td>\n",
       "      <td>200000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TATA CONSULTANCY SERVICES LTD (TCS)</td>\n",
       "      <td>ICICI Bank</td>\n",
       "      <td>Y</td>\n",
       "      <td>13</td>\n",
       "      <td>...</td>\n",
       "      <td>13.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6762.9</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>G</td>\n",
       "      <td>S122</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Male</td>\n",
       "      <td>Panchkula</td>\n",
       "      <td>22500</td>\n",
       "      <td>600000.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ALCHEMIST HOSPITALS LTD</td>\n",
       "      <td>State Bank of India</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S143</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Male</td>\n",
       "      <td>Saharsa</td>\n",
       "      <td>35000</td>\n",
       "      <td>1000000.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>BIHAR GOVERNMENT</td>\n",
       "      <td>State Bank of India</td>\n",
       "      <td>Y</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S143</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Male</td>\n",
       "      <td>Bengaluru</td>\n",
       "      <td>100000</td>\n",
       "      <td>500000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>25000.0</td>\n",
       "      <td>GLOBAL EDGE SOFTWARE</td>\n",
       "      <td>HDFC Bank</td>\n",
       "      <td>Y</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S134</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Gender       City  Monthly_Income  Loan_Amount_Applied  \\\n",
       "0  Female      Delhi           20000             300000.0   \n",
       "1    Male     Mumbai           35000             200000.0   \n",
       "2    Male  Panchkula           22500             600000.0   \n",
       "3    Male    Saharsa           35000            1000000.0   \n",
       "4    Male  Bengaluru          100000             500000.0   \n",
       "\n",
       "   Loan_Tenure_Applied  Existing_EMI                        Employer_Name  \\\n",
       "0                  5.0           0.0                              CYBOSOL   \n",
       "1                  2.0           0.0  TATA CONSULTANCY SERVICES LTD (TCS)   \n",
       "2                  4.0           0.0              ALCHEMIST HOSPITALS LTD   \n",
       "3                  5.0           0.0                     BIHAR GOVERNMENT   \n",
       "4                  2.0       25000.0                 GLOBAL EDGE SOFTWARE   \n",
       "\n",
       "        Salary_Account Mobile_Verified  Var5    ...    Interest_Rate  \\\n",
       "0            HDFC Bank               N     0    ...              NaN   \n",
       "1           ICICI Bank               Y    13    ...            13.25   \n",
       "2  State Bank of India               Y     0    ...              NaN   \n",
       "3  State Bank of India               Y    10    ...              NaN   \n",
       "4            HDFC Bank               Y    17    ...              NaN   \n",
       "\n",
       "   Processing_Fee  EMI_Loan_Submitted  Filled_Form  Device_Type  Var2 Source  \\\n",
       "0             NaN                 NaN            N  Web-browser     G   S122   \n",
       "1             NaN              6762.9            N  Web-browser     G   S122   \n",
       "2             NaN                 NaN            N  Web-browser     B   S143   \n",
       "3             NaN                 NaN            N  Web-browser     B   S143   \n",
       "4             NaN                 NaN            N  Web-browser     B   S134   \n",
       "\n",
       "  Var4 LoggedIn Disbursed  \n",
       "0    1        0         0  \n",
       "1    3        0         0  \n",
       "2    1        0         0  \n",
       "3    3        0         0  \n",
       "4    3        1         0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#导入数据集\n",
    "train_data = pd.read_csv(\"./Train.csv\",encoding='ISO-8859-1').drop([\"ID\",\"Lead_Creation_Date\",\"DOB\"], axis=1)\n",
    "test_data= pd.read_csv(\"./Test.csv\",encoding='ISO-8859-1').drop([\"ID\",\"Lead_Creation_Date\",\"DOB\"], axis=1)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 23 columns):\n",
      "Gender                   87020 non-null object\n",
      "City                     86017 non-null object\n",
      "Monthly_Income           87020 non-null int64\n",
      "Loan_Amount_Applied      86949 non-null float64\n",
      "Loan_Tenure_Applied      86949 non-null float64\n",
      "Existing_EMI             86949 non-null float64\n",
      "Employer_Name            86949 non-null object\n",
      "Salary_Account           75256 non-null object\n",
      "Mobile_Verified          87020 non-null object\n",
      "Var5                     87020 non-null int64\n",
      "Var1                     87020 non-null object\n",
      "Loan_Amount_Submitted    52407 non-null float64\n",
      "Loan_Tenure_Submitted    52407 non-null float64\n",
      "Interest_Rate            27726 non-null float64\n",
      "Processing_Fee           27420 non-null float64\n",
      "EMI_Loan_Submitted       27726 non-null float64\n",
      "Filled_Form              87020 non-null object\n",
      "Device_Type              87020 non-null object\n",
      "Var2                     87020 non-null object\n",
      "Source                   87020 non-null object\n",
      "Var4                     87020 non-null int64\n",
      "LoggedIn                 87020 non-null int64\n",
      "Disbursed                87020 non-null int64\n",
      "dtypes: float64(8), int64(5), object(10)\n",
      "memory usage: 15.3+ MB\n"
     ]
    }
   ],
   "source": [
    "#看起来有很多类别行的属性需要做特征值处理\n",
    "train_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                       0\n",
       "City                      1003\n",
       "Monthly_Income               0\n",
       "Loan_Amount_Applied         71\n",
       "Loan_Tenure_Applied         71\n",
       "Existing_EMI                71\n",
       "Employer_Name               71\n",
       "Salary_Account           11764\n",
       "Mobile_Verified              0\n",
       "Var5                         0\n",
       "Var1                         0\n",
       "Loan_Amount_Submitted    34613\n",
       "Loan_Tenure_Submitted    34613\n",
       "Interest_Rate            59294\n",
       "Processing_Fee           59600\n",
       "EMI_Loan_Submitted       59294\n",
       "Filled_Form                  0\n",
       "Device_Type                  0\n",
       "Var2                         0\n",
       "Source                       0\n",
       "Var4                         0\n",
       "LoggedIn                     0\n",
       "Disbursed                    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### 查看是否有空值\n",
    "train_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                       0\n",
       "City                       398\n",
       "Monthly_Income               0\n",
       "Loan_Amount_Applied         40\n",
       "Loan_Tenure_Applied         40\n",
       "Existing_EMI                40\n",
       "Employer_Name               42\n",
       "Salary_Account            5037\n",
       "Mobile_Verified              0\n",
       "Var5                         0\n",
       "Var1                         0\n",
       "Loan_Amount_Submitted    14922\n",
       "Loan_Tenure_Submitted    14922\n",
       "Interest_Rate            25607\n",
       "Processing_Fee           25746\n",
       "EMI_Loan_Submitted       25607\n",
       "Filled_Form                  0\n",
       "Device_Type                  0\n",
       "Var2                         0\n",
       "Source                       0\n",
       "Var4                         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Monthly_Income</th>\n",
       "      <th>Loan_Amount_Applied</th>\n",
       "      <th>Loan_Tenure_Applied</th>\n",
       "      <th>Existing_EMI</th>\n",
       "      <th>Var5</th>\n",
       "      <th>Loan_Amount_Submitted</th>\n",
       "      <th>Loan_Tenure_Submitted</th>\n",
       "      <th>Interest_Rate</th>\n",
       "      <th>Processing_Fee</th>\n",
       "      <th>EMI_Loan_Submitted</th>\n",
       "      <th>Var4</th>\n",
       "      <th>LoggedIn</th>\n",
       "      <th>Disbursed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>8.702000e+04</td>\n",
       "      <td>8.694900e+04</td>\n",
       "      <td>86949.000000</td>\n",
       "      <td>8.694900e+04</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>5.240700e+04</td>\n",
       "      <td>52407.000000</td>\n",
       "      <td>27726.000000</td>\n",
       "      <td>27420.000000</td>\n",
       "      <td>27726.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.884997e+04</td>\n",
       "      <td>2.302507e+05</td>\n",
       "      <td>2.131399</td>\n",
       "      <td>3.696228e+03</td>\n",
       "      <td>4.961503</td>\n",
       "      <td>3.950106e+05</td>\n",
       "      <td>3.891369</td>\n",
       "      <td>19.197474</td>\n",
       "      <td>5131.150839</td>\n",
       "      <td>10999.528377</td>\n",
       "      <td>2.949805</td>\n",
       "      <td>0.029350</td>\n",
       "      <td>0.014629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.177511e+06</td>\n",
       "      <td>3.542068e+05</td>\n",
       "      <td>2.014193</td>\n",
       "      <td>3.981021e+04</td>\n",
       "      <td>5.670385</td>\n",
       "      <td>3.082481e+05</td>\n",
       "      <td>1.165359</td>\n",
       "      <td>5.834213</td>\n",
       "      <td>4725.837644</td>\n",
       "      <td>7512.323050</td>\n",
       "      <td>1.697720</td>\n",
       "      <td>0.168785</td>\n",
       "      <td>0.120062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>11.990000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>1176.410000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.650000e+04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000e+05</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>15.250000</td>\n",
       "      <td>2000.000000</td>\n",
       "      <td>6491.600000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2.500000e+04</td>\n",
       "      <td>1.000000e+05</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000e+05</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>4000.000000</td>\n",
       "      <td>9392.970000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.000000e+04</td>\n",
       "      <td>3.000000e+05</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.500000e+03</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>5.000000e+05</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>6250.000000</td>\n",
       "      <td>12919.040000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>4.445544e+08</td>\n",
       "      <td>1.000000e+07</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.000000e+07</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>3.000000e+06</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>144748.280000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Monthly_Income  Loan_Amount_Applied  Loan_Tenure_Applied  Existing_EMI  \\\n",
       "count    8.702000e+04         8.694900e+04         86949.000000  8.694900e+04   \n",
       "mean     5.884997e+04         2.302507e+05             2.131399  3.696228e+03   \n",
       "std      2.177511e+06         3.542068e+05             2.014193  3.981021e+04   \n",
       "min      0.000000e+00         0.000000e+00             0.000000  0.000000e+00   \n",
       "25%      1.650000e+04         0.000000e+00             0.000000  0.000000e+00   \n",
       "50%      2.500000e+04         1.000000e+05             2.000000  0.000000e+00   \n",
       "75%      4.000000e+04         3.000000e+05             4.000000  3.500000e+03   \n",
       "max      4.445544e+08         1.000000e+07            10.000000  1.000000e+07   \n",
       "\n",
       "               Var5  Loan_Amount_Submitted  Loan_Tenure_Submitted  \\\n",
       "count  87020.000000           5.240700e+04           52407.000000   \n",
       "mean       4.961503           3.950106e+05               3.891369   \n",
       "std        5.670385           3.082481e+05               1.165359   \n",
       "min        0.000000           5.000000e+04               1.000000   \n",
       "25%        0.000000           2.000000e+05               3.000000   \n",
       "50%        2.000000           3.000000e+05               4.000000   \n",
       "75%       11.000000           5.000000e+05               5.000000   \n",
       "max       18.000000           3.000000e+06               6.000000   \n",
       "\n",
       "       Interest_Rate  Processing_Fee  EMI_Loan_Submitted          Var4  \\\n",
       "count   27726.000000    27420.000000        27726.000000  87020.000000   \n",
       "mean       19.197474     5131.150839        10999.528377      2.949805   \n",
       "std         5.834213     4725.837644         7512.323050      1.697720   \n",
       "min        11.990000      200.000000         1176.410000      0.000000   \n",
       "25%        15.250000     2000.000000         6491.600000      1.000000   \n",
       "50%        18.000000     4000.000000         9392.970000      3.000000   \n",
       "75%        20.000000     6250.000000        12919.040000      5.000000   \n",
       "max        37.000000    50000.000000       144748.280000      7.000000   \n",
       "\n",
       "           LoggedIn     Disbursed  \n",
       "count  87020.000000  87020.000000  \n",
       "mean       0.029350      0.014629  \n",
       "std        0.168785      0.120062  \n",
       "min        0.000000      0.000000  \n",
       "25%        0.000000      0.000000  \n",
       "50%        0.000000      0.000000  \n",
       "75%        0.000000      0.000000  \n",
       "max        1.000000      1.000000  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#对数据值型特征，用常用统计量观察其分布\n",
    "train_data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gender属性的不同取值和出现的次数\n",
      "Male      49848\n",
      "Female    37172\n",
      "Name: Gender, dtype: int64\n",
      "\n",
      "Gender属性的不同取值和出现的次数\n",
      "Male      49848\n",
      "Female    37172\n",
      "Name: Gender, dtype: int64\n",
      "\n",
      "City属性的不同取值和出现的次数\n",
      "Delhi                  12527\n",
      "Bengaluru              10824\n",
      "Mumbai                 10795\n",
      "Hyderabad               7272\n",
      "Chennai                 6916\n",
      "Pune                    5207\n",
      "Kolkata                 2888\n",
      "Ahmedabad               1788\n",
      "Jaipur                  1331\n",
      "Gurgaon                 1212\n",
      "Coimbatore              1147\n",
      "Thane                    905\n",
      "Chandigarh               870\n",
      "Surat                    802\n",
      "Visakhapatnam            764\n",
      "Indore                   734\n",
      "Vadodara                 624\n",
      "Nagpur                   594\n",
      "Lucknow                  580\n",
      "Ghaziabad                560\n",
      "Bhopal                   513\n",
      "Kochi                    492\n",
      "Patna                    461\n",
      "Faridabad                447\n",
      "Madurai                  375\n",
      "Noida                    373\n",
      "Gautam Buddha Nagar      338\n",
      "Dehradun                 314\n",
      "Raipur                   289\n",
      "Bhubaneswar              277\n",
      "                       ...  \n",
      "SAYAN                      1\n",
      "SOMNATH JUNAGADHA          1\n",
      "Surendra Nagar             1\n",
      "Doda                       1\n",
      "Lohit                      1\n",
      "Pulwama                    1\n",
      "Dantewada                  1\n",
      "Muktsar                    1\n",
      "Malabar                    1\n",
      "Lakhisarai                 1\n",
      "Baksa                      1\n",
      "Hazaribagh                 1\n",
      "Raisen                     1\n",
      "Rampur                     1\n",
      "Jashpur                    1\n",
      "Narayanpur                 1\n",
      "VIRPUR                     1\n",
      "Imphal East                1\n",
      "IDAR                       1\n",
      "Chandel                    1\n",
      "Gadwal                     1\n",
      "Narsinghpur                1\n",
      "Umaria                     1\n",
      "Anjaw                      1\n",
      "KHAMBHAT                   1\n",
      "Khagaria                   1\n",
      "Shahpura                   1\n",
      "Siruguppa                  1\n",
      "CHOTILA                    1\n",
      "Seoni                      1\n",
      "Name: City, Length: 697, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#可以看到空值很多，需要对空值做处理\n",
    "categorical_features = ['Gender','Gender','City']\n",
    "for col in categorical_features:\n",
    "    print('\\n%s属性的不同取值和出现的次数'%col)\n",
    "    print(train_data[col].value_counts())\n",
    "    #train_data[col] = train_data[col].astype('object')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#数据清洗以及编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#去掉重复的数据\n",
    "train_data = train_data.drop_duplicates()\n",
    "test_data=test_data.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到去重之前 train_data 是87020 条记录，去重之后 85920， 有优化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                   0\n",
       "City                     0\n",
       "Monthly_Income           0\n",
       "Loan_Amount_Applied      0\n",
       "Loan_Tenure_Applied      0\n",
       "Existing_EMI             0\n",
       "Employer_Name            0\n",
       "Salary_Account           0\n",
       "Mobile_Verified          0\n",
       "Var5                     0\n",
       "Var1                     0\n",
       "Loan_Amount_Submitted    0\n",
       "Loan_Tenure_Submitted    0\n",
       "Interest_Rate            0\n",
       "Processing_Fee           0\n",
       "EMI_Loan_Submitted       0\n",
       "Filled_Form              0\n",
       "Device_Type              0\n",
       "Var2                     0\n",
       "Source                   0\n",
       "Var4                     0\n",
       "LoggedIn                 0\n",
       "Disbursed                0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data=train_data.fillna(method = 'backfill', axis = 0)\n",
    "#train_data.interpolate(method = 'linear', axis = 0)\n",
    "test_data=test_data.fillna(method = 'backfill', axis = 0)\n",
    "train_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#目前看来有一条Salary_Account的记录没有得到清除，由于这个数据样本比较小，可以删除此列。\n",
    "train_data=train_data.dropna(axis = 0)\n",
    "test_data=test_data.dropna(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.isnull().sum().sum()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get_dummies()可以直接对字符型变量进行one-hot编码，对于特征值变量做特征编码，日期处理比较麻烦由于时间问题，暂时不处理会丢弃\n",
    "for col in train_data.columns :\n",
    "    if train_data[col].dtype=='object':\n",
    "        train_data[col] = pd.get_dummies(train_data[col])\n",
    "        train_data[col]\n",
    "train_data.head()\n",
    "\n",
    "for col in test_data.columns :\n",
    "    if test_data[col].dtype=='object':\n",
    "        test_data[col] = pd.get_dummies(test_data[col])\n",
    "        test_data[col]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型建立，本章基于xgboost来建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "#测试数据和训练数据进一步分离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = ['Disbursed']\n",
    "y_train=train_data[categorical_features]\n",
    "X_train=train_data.drop(['LoggedIn','Disbursed'], axis=1)\n",
    "X_train_array = np.array(X_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认参数，此时学习率为0.1，比较大，观察弱分类数目的大致范围 （采用默认参数配置，看看模型是过拟合还是欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def modelfit(alg, X_train_value, y_train_value, cv_folds=None, early_stopping_rounds=10):\n",
    "    xgb_param = alg.get_xgb_params()    \n",
    "    #直接调用xgboost，而非sklarn的wrapper类\n",
    "    xgtrain = xgb.DMatrix(X_train_value, label = y_train_value)\n",
    "    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "             metrics='logloss', early_stopping_rounds=early_stopping_rounds)\n",
    "  \n",
    "    cvresult.to_csv('1_nestimators.csv', index_label = 'n_estimators')\n",
    "    \n",
    "    #最佳参数n_estimators\n",
    "    n_estimators = cvresult.shape[0]\n",
    "    \n",
    "    # 采用交叉验证得到的最佳参数n_estimators，训练模型\n",
    "    alg.set_params(n_estimators = n_estimators)\n",
    "    alg.fit(X_train_value, y_train_value, eval_metric='logloss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "xgb1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators,时间关系 n_estimators 以及depth都比较小\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb1, X_train_array, y_train, cv_folds = kfold)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "#n_estimators调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XmcHWWd7/HP95xOp9PZOktn7eyEQICwGDZRB8UFdAyIiDDgiKNynTtcGDcG53qZO87ADDrD6IxclXGQmVFA1FGjMriwuKBgEgKEJARC9pClyb73cn73j6puDk13upP06eru832/XvXqU089p+pXVcn5nXqeOvUoIjAzMwPIZR2AmZn1Hk4KZmbWyknBzMxaOSmYmVkrJwUzM2vlpGBmZq2cFMyKSPpLSV/POg6zrDgp9DGShkhaI+mqorKhktZJuqyobK6kH0vaIWmnpGWSbpE0Il1+jaRmSXvTaZWkPy1x7OdL2lDKbRyJ9uKJiFsj4iMl2t4aSW8txbpLoafOV187Lv2dk0IfExF7gf8BfFFSbVr8eWBhRHwXQNLrgUeBx4ATIqIGuBBoAk4tWt3vImJIRAwB3gt8XtLpPbMndiQkVWQdg5WJiPDUByfgbuBe4HxgGzCuaNlvgH/p5P3XAL9pU/Z74I+K5ucBS4GdJEnmxKJlJ6ZlO9M684qWvRNYBuwBNgKfAgYDB4ACsDedJnSwX3cAP0nf/wQwowvH4wTg58B2YAVw+dHEA/xf4Jvp+6YCAXwIWA/sAD4GnAk8k+77l4u2MwN4OD0fLwPfAmrSZf+ZbutAuq0bu3CM1wB/kW7rEFCRzm9M92UFcEE7x+JsYDOQLyp7D/BM+vosYCGwG9gC3N7BMT0f2NDBsuHAfwD1wFrgs0AuXZYH/jE9BquB69LjWNHButYAb+1g2UeBlel5nd/ybwYQ8E/A1nQ/lgAnd3S+s/7/2pemzAPwdJQnDkYAm9L/eB8qKh8MNAPnd/L+ayhKCukH3U7g+HT+eGAf8DZgAHBj+p+zMp1fCfxlOv+W9D/grPS9m4A3FsV5Rvq6ww+ZojjuJvlQPSv9EPwWcF8n7xlM8qH9ofQ9p6fHZfaRxkP7SeGrQBXwduAg8ANgDDAx/VD6g7T+cenxGgjUAr8Cvli07ld9+B3uGBfVfwqYBAwCZqX7OaEovnYTJvAi8Lai+e8AN6Wvfwd8IH09BDing3V0eL5IEsIPgaFpHM8DH06XfYzkQ7kuPd6/4CiSQvrv6mXgjPSY/gvwq3TZO4BFQA1JgjgRGH+48+2pa5Obj/qoiNhB8g2zGvivokUjSJoFN7cUSPp82q+wT9Jni+qek5bvIblK+E/ghXTZ+4GfRMTPI6IR+AeSD6bXA+eQfJj8fUQ0RMTDwI+BK9P3NgKzJQ2LiB0R8eQR7t73I+L3EdFEkhRO66T+HwJrIuIbEdEUEYuB7wHv66Z4/iYiDkbEz0g+xO+NiK0RsRH4NUkSIiJWpsfrUETUA7cDf3CY9R7uGLf454hYHxEHSJL9wHRfBkTEmoh4sYN130t6PiQNJfn2fG/R8ThO0uiI2BsRjx/JwZCUB64APhMReyJiDcmVwQfSKpcDX4qIDem/078/kvUXuQq4KyKejIhDwGeAcyVNTfdhKMkVoiJieURsKtq/YznfZc1JoY+SdDXJN7RfALcVLdpB0kwxvqUgIm6MpF/h+yTfpFs8HhE1ETEUGAecBNyaLptA0izQso4CybfUiemy9WlZi7XpMkj6J94JrJX0S0nnHuHubS56vZ8kAR3OFODsNMHtlLST5ANlXDfFs6Xo9YF25ocASBor6T5JGyXtBr4JjD7Meg93jFusL1q+EvhzkquZrem2JnSw7nuASyUNBC4FnoyIlm19mOQq5TlJCyT94WFibM9okiubtUVlxed/QnHcbV4fibbHZy/JVeTE9IvIl0maGrdKulPSsLTqsZ7vsuak0AdJGkPSnvpRkk7nyyW9ESAi9pG0w196JOuMiC0k367fnRa9RPJh27JNkTRjbEyXTZJU/O9ncrqMiFgQEReTNLH8ALi/ZTNHEtMRWA/8Mk1wLdOQiPjTHo7n1nSdp0TEMOBqkqaNFm23d7hj3O57IuKeiHhD+r7g1V8IiustI/lAvQj4I5Ik0bLshYi4kuR43AZ8V9Lgru8mL5N8G59SVNZ6/kmab+qKlk06gnUXa3t8BgOjeOXf2T9HxOuA2SRJ7tNpeUfn27rASaFv+jLwg4h4JL1kvhH41/RbIen8n0i6KU0gSKoDpnW0QkmjSDojl6ZF9wPvknSBpAHAJ0k6O39LknT2AzdKGiDpfJJkcp+kSklXSRqeNonsJrlygeQb9ihJw7vpOLT4MXC8pA+k8QyQdKakE3s4nqEknci7JE0k/ZAqsgWYXjR/uGP8GpJmSXpLep4P8kpHeUfuAW4A3kTSp9Cynqsl1aZXJjvT4g7XI6mqeErr3g/ckt4OPQX4BMmVUct+3SBpoqQaks7xzgxos50KkuauD0k6Ld3nW4EnImJNen7PTo/bvvR4FDo539YVWXdqeDqyCbiE5BtUTZvyh4FbiubPBh4g+U+/E3gWuAUYlS6/hqSNuuXOm60k/wnHFK3jPSQdhruAXwInFS07KS3bldZ5T1peCTxI0oy1G1gAvKHofXeRNAHspOO7j/62aP58OumcTuvNIrljqT5d/8MkfRFHFA/tdzRXFNXfQFEnPskH4WeLjsmi9Hg+RfIhv6Go7sXAunRbn+rCMV7Dqzum55D0/ewhuRvnx+0dw6L6k0k+EH/Spvyb6fneS/Il4JIO3n9+uv9tp+NI+q6+mR7v9cDNvHL3UQXJlew2kruPPk5yZaEOtrOmnW38bbrsYySd5i37W5eWX0ByV9ZeXrnTa0hn59tT55PSA2xmVhKSLgK+GhFTOq1smXPzkZl1K0mDJL1TUkXajPZXJDc5WB/gKwXrE9KO9P9ub1kkv8i2XkJSNUlT2Akk/R4/AW6IiN2ZBmZd4qRgZmat3HxkZmat+txDtkaPHh1Tp07NOgwzsz5l0aJFL0dEbWf1+lxSmDp1KgsXLsw6DDOzPkXS2s5rufnIzMyKOCmYmVkrJwUzM2vlpGBmZq2cFMzMrJWTgpmZtXJSMDOzViVNCpIulLRC0kpJN3VQ53JJyyQtlXRPe3W6w9rli/j9975IFPxodTOzjpQsKaTjuN5BMvLTbOBKSbPb1JlJMu7qeRFxEslQgyWxadF8zlryV+zZvaNUmzAz6/NKeaVwFrAyIlZFRANwH8kgI8U+CtwRyeDeRMTWUgUzYEQyIuD2TatLtQkzsz6vlElhIq8esHsDrx6QHJJxVY+X9JikxyVd2N6KJF0raaGkhfX19UcVzODRkwHYvXnNUb3fzKwcZN3RXAHMJBn270qScYZr2laKiDsjYm5EzK2t7fR5Tu0aPi4Z9Ong9vWd1DQzK1+lTAobgUlF83VpWbENwPyIaIyI1cDzJEmi240eP5VCiOadbUMwM7MWpUwKC4CZkqZJqgSuAOa3qfMDkqsEJI0maU5aVYpgBlQOZJtqyO99qRSrNzPrF0qWFCKiCbgO+CmwHLg/IpZK+pykeWm1nwLbJC0DHgE+HRHbShXTjopaqg5sKdXqzcz6vJKOpxARDwAPtCm7ueh1AJ9Ip5LbN3AMIw506ZHiZmZlKeuO5h7VUD2OUc0vZx2GmVmvVVZJIYZOYKgOsGfX9qxDMTPrlcoqKVSMbPkB25pM4zAz663KKilUt/yAbcuabAMxM+ulyiop1IydCsDBbf4Bm5lZe8oqKYwan1wpNPkHbGZm7SqrpDCwqpqX8Q/YzMw6UlZJAWBHfjQD92/OOgwzs16p7JLC3oFjGNZQsid0m5n1aWWXFBqqxzGq4B+wmZm1p+ySQmFYHcPYx749O7MOxcys1ym7pDCgJhnnZ9umNZnGYWbWG5VdUhiU/oBt12Y/GM/MrK2ySwo16QhsB7atyzgSM7Pep+ySwqjxUwFo3rkh20DMzHqhsksKVYMGs51h5PZuyjoUM7Nep+ySAsD2fK1/wGZm1o6yTAp7B45h2CEPy2lm1lZZJoVD1eMZ6R+wmZm9RlkmhcLQCdSw1z9gMzNroyyTQmXtdAC2rluRcSRmZr1LWSaFYeNnArBz4wsZR2Jm1ruUZVIYM3kWAIfqV2UciZlZ71KWSWHYiFr2MAjtWJN1KGZmvUpZJgXlcmzNj6dqn8dqNjMrVpZJAWBX1URqDnlYTjOzYmWbFBqGTmZs8xYKzc1Zh2Jm1muUNClIulDSCkkrJd3UzvJrJNVLeiqdPlLKeF617ZFTqVIj27a4CcnMrEXJkoKkPHAHcBEwG7hS0ux2qn47Ik5Lp6+XKp62Bo2ZAcDL6/1bBTOzFqW8UjgLWBkRqyKiAbgPuLiE2zsiI+qOB2DvppUZR2Jm1nuUMilMBIrbZjakZW29V9Izkr4raVJ7K5J0raSFkhbW19d3S3Bj646jEKJp2+puWZ+ZWX+QdUfzj4CpETEH+Dnw7+1Viog7I2JuRMytra3tlg1XVg1iq0ZRsdvDcpqZtShlUtgIFH/zr0vLWkXEtog4lM5+HXhdCeN5jW2V4xmyf2PnFc3MykQpk8ICYKakaZIqgSuA+cUVJI0vmp0HLC9hPK+xr3oSoxo9ApuZWYuSJYWIaAKuA35K8mF/f0QslfQ5SfPSatdLWirpaeB64JpSxdOe5uGTGcN2Du7f25ObNTPrtSpKufKIeAB4oE3ZzUWvPwN8ppQxHM6A0dNhLWxZ9zxTTjgjqzDMzHqNrDuaMzVk/HEA7Hzp+YwjMTPrHco6KdROSn6rcGCLH6FtZgZlnhRG1k5kfwwEP0LbzAwo86SgXI4t+XEM3OvnH5mZQZknBYCdVROpOehHaJuZgZMCu/YdYGzzS36EtpkZTgoMOuldVKuBrS/5GUhmZmWfFIZMTJ7mXb96ScaRmJllr+yTwtgZpwCwb2OPPmHDzKxXKvukMGpMHbupRtteyDoUM7PMlX1SUC7HpopJDN7jH7CZmZV9UgDYPWQ6Yw6tyzoMM7PMOSkATSNnMobt7N65LetQzMwy5aQAVI2bBcDmVb4DyczKm5MCMGrqyQDsWrc040jMzLLlpACMn3oijZGnaeuKrEMxM8uUkwIwoHIgL+XHU7XrxaxDMTPLlJNCanvVFEYeWJt1GGZmmXJSSB2smcH45pdobDiUdShmZplxUkhVjJlFpZrZtPa5rEMxM8uMk0Jq+KSTANjmB+OZWRlzUkiNmzEHgIObfQeSmZUvJ4XUsJpR1DOCiu1+MJ6ZlS8nhSJbKyczbK8H2zGz8uWkUGTv0OlMaFxLFApZh2JmlgknhSIadxJDdYBN69yEZGblyUmhyLBpZwCw+fkFGUdiZpYNJ4Uik0+YSyHEoQ1PZx2KmVkmSpoUJF0oaYWklZJuOky990oKSXNLGU9nqocMZ2NuPAO3ebxmMytPJUsKkvLAHcBFwGzgSkmz26k3FLgBeKJUsRyJ+sHHMXb/81mHYWaWiVJeKZwFrIyIVRHRANwHXNxOvb8BbgMOljCWLmsYNZuJsYU9u7ZnHYqZWY8rZVKYCKwvmt+QlrWSdAYwKSJ+crgVSbpW0kJJC+vr67s/0iKDJp0GwMbnFpZ0O2ZmvVFmHc2ScsDtwCc7qxsRd0bE3IiYW1tbW9K4xh2fdGvsWrO4pNsxM+uNSpkUNgKTiubr0rIWQ4GTgUclrQHOAeZn3dk8pm4GuxgMW57NMgwzs0yUMiksAGZKmiapErgCmN+yMCJ2RcToiJgaEVOBx4F5EZFpu41yOTZUzqBmtx+MZ2blp9OkIGmGpIHp6/MlXS+pprP3RUQTcB3wU2A5cH9ELJX0OUnzjjXwUtpTcwKTGtfQ3NSUdShmZj2qK1cK3wOaJR0H3EnSJHRPV1YeEQ9ExPERMSMibknLbo6I+e3UPT/rq4QWufGnUK1DbFy9NOtQzMx6VFeSQiH91v8e4F8i4tPA+NKGla2R05PHXdS/sCjjSMzMelZXkkKjpCuBDwI/TssGlC6k7E2adTpNkaNh4zNZh2Jm1qO6khQ+BJwL3BIRqyVNA/6ztGFla2DVYNbnJzFoux93YWblpaKzChGxDLgeQNIIYGhE3FbqwLK2jeFM3v9c1mGYmfWortx99KikYZJGAk8C/yrp9tKHlq2mmRcyRjvZsnFV1qGYmfWYrjQfDY+I3cClwH9ExNnAW0sbVvZGzDwXgI1Lfp1xJGZmPacrSaFC0njgcl7paO73pp58Dg1RwaG1v886FDOzHtOVpPA5kh+gvRgRCyRNB/r9eJUDq6pZM2A6w7Z5wB0zKx+dJoWI+E5EzImIP03nV0XEe0sfWvZ2jJjD1EPP09TYmHUoZmY9oisdzXWSvi9pazp9T1JdTwSXtfykuQzWIdY+92TWoZiZ9YiuNB99g+RBdhPS6UdpWb83bvYbAHh5xWMZR2Jm1jO6khRqI+IbEdGUTncDpR3UoJeYOP0kdjIEbewVj2QyMyu5riSFbZKulpRPp6uBbaUOrDdQLse6qhOp3eWxFcysPHQlKfwJye2om4FNwGXANSWMqVfZX3saU5rXecxmMysLXbn7aG1EzIuI2ogYExGXAGVx9xFA9fSzySlYu8T9CmbW/x3tyGuf6NYoerEpc94IwJ4XH884EjOz0jvapKBujaIXGz5qHBs0nqoti7MOxcys5I42KUS3RtHLbRp2ClP2LyEKhaxDMTMrqQ6TgqQ9kna3M+0h+b1C2YjJ5zGS3ax9/qmsQzEzK6kOk0JEDI2IYe1MQyOi03EY+pOJpyUPhd3y9C8yjsTMrLSOtvmorEyYNputjCS/4bdZh2JmVlJOCl2gXI51w85g6p7F7lcws37NSaGLCpPPYzQ7WffCM1mHYmZWMk4KXTT+1KRfYbP7FcysH+u0wzi926jtLai7gIXAJyOiLAYxrptxMvWMIL/e/Qpm1n915S6iLwIbgHtIfrR2BTADeBK4Czi/VMH1JsrlWDf0dCan/QrK+SLLzPqfrnyyzYuIr0XEnojYHRF3Au+IiG8DI0ocX6/SPPk8xrCd9S/6qalm1j91JSnsl3S5pFw6XQ4cTJeV1S+bx6X9CpuefijjSMzMSqMrSeEq4APA1nT6AHC1pEHAdYd7o6QLJa2QtFLSTe0s/5ikJZKekvQbSbOPYh96zKTj5rCNGvLrfpN1KGZmJdFpn0LakfzuDhZ3+OkoKQ/cAbyNpE9igaT5EbGsqNo9EfHVtP484Hbgwi7G3uOUy7Fm2OuYunshheZmcvl81iGZmXWrTq8UJNVJ+r6kren0PUl1XVj3WcDKiFgVEQ3AfcDFxRUiYnfR7GD6QHNUzLiA0ezkxSW+C8nM+p+uNB99A5hP8hC8CcCP0rLOTATWF81vSMteRdKfSXoR+DxwfXsrknStpIWSFtbX13dh06Uz/Zwkr728+MeZxmFmVgpdSQq1EfGNiGhKp7uB2u4KICLuiIgZwF8An+2gzp0RMTci5tbWdtumj8rIsXW8UDGTmpd+mWkcZmal0JWksE3S1ZLy6XQ1sK0L79sITCqar0vLOnIfcEkX1pu5bePexPENz7Hz5S1Zh2Jm1q26khT+BLgc2AxsAi4DrunC+xYAMyVNk1RJ8qO3+cUVJM0smn0X8EIX1pu5Eae9i7yClY/P77yymVkf0mlSiIi1ETEvImojYkxEXAK8twvvayK5ZfWnwHLg/ohYKulz6Z1GANdJWirpKZJxnz949LvSc4477Q/YwVDihZ9nHYqZWbc62sFyPkHy+IvDiogHgAfalN1c9PqGo9x+pvIVFbw47Gxm7Pqdb001s37laB/go26Noi867q2MZDcrn34s60jMzLrN0SaFXv97glKbce7FFEK8/JRvTTWz/qPDpCBpj6Td7Ux7SH6vUNZG1E7g+ahj5JoHOq9sZtZHdJgUImJoRAxrZxoaEUfbF9Gv7DzuPZyQW89La1ZkHYqZWbfwoADHYPJ5VwCw9jffzjgSM7Pu4aRwDCbMOIkX89OpWesmJDPrH5wUjlH9pIs4sXE5mzeUxaikZtbPOSkco4nnvR+A1b+6L+NIzMyOnZPCMZo081RW56YybPVPsg7FzOyYOSl0gy117+DEhqVsfWld1qGYmR0TJ4VuMP71V5BTsOrXbkIys77NSaEbTDnhDFbGRCqW/VfWoZiZHRMnhW6ybcZ7mKvlrH9xadahmJkdNSeFbjL9gg9TCLH+kbuyDsXM7Kg5KXST2onTWTroDKZt+CHNzc1Zh2NmdlScFLpR4ylXMp56lv3Wt6eaWd/kpNCNZr/5SvYwiIML/jPrUMzMjoqTQjeqqh7Cc6Pexsm7HmX3ru1Zh2NmdsScFLrZ8HOvYZAaWP4LXy2YWd/jpNDNZp7xZtbl6qhZ/k0iyn6AOjPrY5wUuplyOTYffxWzmp7nuUWPZh2OmdkRcVIogZPe+TH2RRV7f3VH1qGYmR0RJ4USGDxsJM+OeTen7nqY+k1+SJ6Z9R1OCiUy8e3XU6lmXnjgX7IOxcysy5wUSqRu5hyWDDqT49ffz6FDB7IOx8ysS5wUSums/8FodvL0T/8960jMzLrESaGETnrTpazPTWTkU1+j0FzIOhwzs06VNClIulDSCkkrJd3UzvJPSFom6RlJD0maUsp4eloun2frnD/luMIqnnrk/qzDMTPrVMmSgqQ8cAdwETAbuFLS7DbVFgNzI2IO8F3g86WKJyunvvNaNqmW6sdvJwq+WjCz3q2UVwpnASsjYlVENAD3ARcXV4iIRyJifzr7OFBXwngyUVE5kA2zP8YJTSt45tc/zDocM7PDKmVSmAisL5rfkJZ15MPAf5cwnsyc+u4/o56R5H/zj370hZn1ar2io1nS1cBc4AsdLL9W0kJJC+vr63s2uG5QWTWI1Sd8hJMbl/Ds7x7MOhwzsw6VMilsBCYVzdelZa8i6a3A/wbmRcSh9lYUEXdGxNyImFtbW1uSYEttzrzr2UYNuUf+xn0LZtZrlTIpLABmSpomqRK4AphfXEHS6cDXSBLC1hLGkrmq6qGsOfl/cVLjUhb97FtZh2Nm1q6SJYWIaAKuA34KLAfuj4ilkj4naV5a7QvAEOA7kp6SNL+D1fULp11yA+tydYx54lYOHTqYdThmZq9R0j6FiHggIo6PiBkRcUtadnNEzE9fvzUixkbEaek07/Br7NvyFQPY9cabmRwvsfB7/5R1OGZmr9ErOprLySnnX86ygady4vN3sGvHtqzDMTN7FSeFniYx6F23MpI9PHvv/846GjOzV3FSyMC0OW9g4ah5nL3l26x46rGswzEza+WkkJFZH7id3RoKP7qBxsbGrMMxMwOcFDIztKaW9Wf9H2Y1v8AT9/e7Rz6ZWR/lpJChORd+mGcHncnpz/8zL61dmXU4ZmZOCllSLkftFV8mR4H6b32U5ubmrEMyszLnpJCxsVNOYNkpN3Jqw5P87t5bsw7HzMqck0IvcMaln2TJ4HM484Uv8cKS32cdjpmVMSeFXkC5HJOvuYt9qib3/Y9yYP/+zt9kZlYCTgq9xPDaiWw+/x+ZUVjDT77wQY+7YGaZcFLoRWaf/z4WTvxjLouf8dj9t2cdjpmVISeFXuaMD/0TSwfN5cxlt/LM4z/POhwzKzNOCr1MrqKCydfey7bcaMY/+FE2rluVdUhmVkacFHqhoSPG0Pz+bzKY/Ry4+1K2b+t7Q5CaWd/kpNBLTTrhTNa/7U6mNK9j41ffw759e7MOyczKgJNCLzbrvEt47pzPc0rjEpZ++QoaGvzgPDMrLSeFXu6Uiz7C4hM/zVkHfs0Tf/sWDjUcyjokM+vHnBT6gNPf/1kWH3cdb8w9w+IvXs7Bgx7f2cxKw0mhjzj96ltYPOsTnLP/UZZ86b3sP+BfPZtZ93NS6ENOv/KvWDz7LzjzwG9Yefvbqa/fnHVIZtbPOCn0Madf/pc8e/YXOKFhOfv+31tY9fzSrEMys37ESaEPOvmia9nw7nsYEbsY/q0LWfjoj7IOycz6CSeFPmr63Hdw8I8f5EB+GKc/8gEe+fpnaGxqyjosM+vjnBT6sLHTT2H0Jx5j6Yi38OYN/4+nv3ARL21cl3VYZtaHOSn0cVVDaphzw/d4Zs5nmXPwSQbe+QZ+/aO7/ehtMzsqTgr9gcScSz/N9qt+xu4Bo3njohv47T+8j40b12cdmZn1MU4K/ci441/HlL/4HYunXctZex+m+s5zeOibt3HwUEPWoZlZH1HSpCDpQkkrJK2UdFM7y98k6UlJTZIuK2Us5SI3YCCnf/AL7Pjjh6ivnsEFK29l9W3n8tgvfkCh4CYlMzu8kiUFSXngDuAiYDZwpaTZbaqtA64B7ilVHOVqzIzTOf7GX7Li9bczOnZw3m8+yKK/u4AFj//S/Q1m1qFSXimcBayMiFUR0QDcB1xcXCEi1kTEM0ChhHGUL4lZb/8wI296lmdnf5KZDc9x5oPz+P3fvYPf/fK/feVgZq9RyqQwESju6dyQlh0xSddKWihpYX29B5w5UvmB1Zx8+c0M+tQSlsz8n5zYsJRzH7mCp295Aw//4C727D+QdYhm1kv0iY7miLgzIuZGxNza2tqsw+mzBg4dxSlX/R3VNy5j6cmfpi628JanPs6e207mZ1+7kWUrVrhpyazMVZRw3RuBSUXzdWmZZayiejgnXfZZaL6J1b/9Dk2/+ypv3/Q1mu+5kwUDzmD38ZdxwpveR904J2CzclPKpLAAmClpGkkyuAL4oxJuz45UvoJpb7wS3nglezY+x9qHv8701d9n9LLPcHDpzfx24Fz2THsnk8+6mFnTJpPLKeuIzazEVMrmAknvBL4I5IG7IuIWSZ8DFkbEfElnAt8HRgAHgc0RcdLh1jl37txYuHBhyWIue4Vmti59lPonvs34l37OyMJ2mkMs0Sxeqn0DVbMuYPbr3sS4EUOyjtTMjoCkRRExt9N6fa0N2UmhBxUK7Fj5OJsX/JDB6x5i8qEXANgdg3i24mS2jzqDgdNfz6STzmXGhFoG5PtEF5VZWXJSsG5X2FPPxqceZO/yh6mpX8D4xuTmsqbIsZI6NlQdz/6RJ1E54WRGzXgdM6ZMYuTgyozALYOZAAAK90lEQVSjNjNwUrAeEPteZsvSX7Fr5RPktzxN7Z7lDC/sbF2+JWpYqzq2VU+nYfh08qNnMHTCLEbVzaBu1DCGDxqA5H4Ks57gpGA9L4LYu4Wdqxez/cVFNG99jqqdK6k9uIZB8cpvIZpDbGYkm6hlZ+VYDlSNo2nIRDR8ApUjJlI9qo5ho8dTO2wwo4ZUUl1ZyvshzMqDk4L1HhGwdyv7Nz/P9vXPcbB+FbFjLZV7NjD40GZqml6mguZXvaUQYjtDeTmGs0PD2JsfwYEBI2gcWENT5XCiagQMqiFXXUNF9Qgqh4ykcsgIqgcNZsigAQwZWMGQgRUMGpCnqjJHZT7nqxIra11NCv4KZqUnwdCxVA8dS/XMN752eaEZ9m6laddL7K1fx96XN9C4azOFPVsYuK+eukPbqGpYzeCGxVQf2nfYTTVEnr0MYm8MYjvV7Gcg+2MgB6jiUK6KxlwVjflBNOWqaK4YRCFfRVRUUcgPggFVRL6KqBiIKiqhYiBKXyd/q8jlB6ABA8lXDEAVleTzFeRzOSpyIp/TK3/zelV5PidyavkLOaV1JHK55G8+JySS+hLSK3VzErkcretRcblwwrNu46Rg2cvlYdh4KoaNp2bS66g5XN3mJji4Ew7sgIO7aNy3g4N7ttG4byeN+3bSvH8HhYO70cE9DGvYQ03jfnJN+6lo2ka++SADCgcY0HyQAU0HyTUc+1VyIUQjeRqpoIk8jeRpooKmaHmdp5kczeRpIkeBHA3kKZCjKZL5tnWa0/kCOZojRwElr9OpQFLWWudVdZNl5PI0hghELlfBoYJoRgwcUEGQIxAoWW8gQulfcoAIpa+ltFzJetWyLJ8cAKX1i5ej1rJQ8rqQrr+4XC2xtrxHr8TV8heElGtdTxITretBuTb1kzvgXtkGrfEn+0VRHLRuM/mXULyeVwi1FhUvKi5PliUzOSUhtswfrjWmq3VaNnP1uVN486wxHdbtDk4K1rfkK2Dw6GQCBqTTEYuApkPQuB+aDkLjgWRqPpSUNx2EpoZ0voFC0yGaGw9RaDxIoamRQnMj0dSQ/j1ENDdCoZGKpiYqmhuJQlNa1pROjVAooEITRHNydRRNqPVvE4pmiGZUaEbRhCJJByokKUBRSOsEuWhOlkVSnuvsmZItdwsXf/b0rZbjHlVIE0eBV5JicTJ5JUnRmmST123rU5QcKXpPO+ssTppp7Zb3tiTd7WM/DrP+pKT77qRg5UmCAVXJ1AU5evmDwiIgCslUSJLLK6/blEWkr5OPIgppf05LPeK1dVrfl76maHuvqVMoWkccfn2t22+vfqGLr3lt+ZH+bbPdXPo696p9LX4f7e/La+oV1X9VGZ3E27Z+st6pM6d037+ZDjgpmPUHEigP5CF/VNdOZkAv//JjZmY9y0nBzMxaOSmYmVkrJwUzM2vlpGBmZq2cFMzMrJWTgpmZtXJSMDOzVn3uKamS6oG1R/n20cDL3RhOb+Z97b/KaX+9r91nSkTUdlapzyWFYyFpYVceHdsfeF/7r3LaX+9rz3PzkZmZtXJSMDOzVuWWFO7MOoAe5H3tv8ppf72vPays+hTMzOzwyu1KwczMDsNJwczMWpVNUpB0oaQVklZKuinreLqTpEmSHpG0TNJSSTek5SMl/VzSC+nfEVnH2l0k5SUtlvTjdH6apCfS8/ttSZVZx9gdJNVI+q6k5yQtl3Rufz2vkj6e/vt9VtK9kqr6y3mVdJekrZKeLSpr9zwq8c/pPj8j6YyejLUskoKkPHAHcBEwG7hS0uxso+pWTcAnI2I2cA7wZ+n+3QQ8FBEzgYfS+f7iBmB50fxtwD9FxHHADuDDmUTV/b4EPBgRJwCnkuxzvzuvkiYC1wNzI+JkIA9cQf85r3cDF7Yp6+g8XgTMTKdrga/0UIxAmSQF4CxgZUSsiogG4D7g4oxj6jYRsSkinkxf7yH54JhIso//nlb7d+CSbCLsXpLqgHcBX0/nBbwF+G5apV/sq6ThwJuAfwOIiIaI2Ek/Pa8kwwMPklQBVAOb6CfnNSJ+BWxvU9zRebwY+I9IPA7USBrfM5GWT1KYCKwvmt+QlvU7kqYCpwNPAGMjYlO6aDMwNqOwutsXgRuBdNR2RgE7I6Ipne8v53caUA98I20q+7qkwfTD8xoRG4F/ANaRJINdwCL653lt0dF5zPTzqlySQlmQNAT4HvDnEbG7eFkk9x73+fuPJf0hsDUiFmUdSw+oAM4AvhIRpwP7aNNU1I/O6wiSb8jTgAnAYF7b3NJv9abzWC5JYSMwqWi+Li3rNyQNIEkI34qI/0qLt7RcdqZ/t2YVXzc6D5gnaQ1JM+BbSNrda9JmB+g/53cDsCEinkjnv0uSJPrjeX0rsDoi6iOiEfgvknPdH89ri47OY6afV+WSFBYAM9M7GSpJOrDmZxxTt0nb1P8NWB4Rtxctmg98MH39QeCHPR1bd4uIz0REXURMJTmPD0fEVcAjwGVptf6yr5uB9ZJmpUUXAMvoh+eVpNnoHEnV6b/nln3td+e1SEfncT7wx+ldSOcAu4qamUqubH7RLOmdJG3ReeCuiLgl45C6jaQ3AL8GlvBKO/tfkvQr3A9MJnnc+OUR0bazq8+SdD7wqYj4Q0nTSa4cRgKLgasj4lCW8XUHSaeRdKhXAquAD5F8met351XSXwPvJ7mbbjHwEZK29D5/XiXdC5xP8njsLcBfAT+gnfOYJsUvkzSf7Qc+FBELeyzWckkKZmbWuXJpPjIzsy5wUjAzs1ZOCmZm1spJwczMWjkpmJlZKycFMzNr5aRg1gWSTkt/69IyP6+7HsEu6c8lVXfHusyOlX+nYNYFkq4heazzdSVY95p03S8fwXvyEdHc3bGY+UrB+hVJU9PBaP41HbDlZ5IGdVB3hqQHJS2S9GtJJ6Tl70sHenla0q/SR6N8Dni/pKckvV/SNZK+nNa/W9JXJD0uaZWk89NBVZZLurtoe1+RtDCN66/TsutJHgD3iKRH0rIrJS1JY7it6P17Jf2jpKeBcyX9vZKBlZ6R9A+lOaJWdiLCk6d+MwFTSR6TcFo6fz/JoxHaq/sQMDN9fTbJc5QgeVzIxPR1Tfr3GuDLRe9tnScZQOU+QCRP+twNnELypWtRUSwj07954FFgTjq/Bhidvp5A8hygWpKnpD4MXJIuC5JHIUDyuPAVvHK1X5P1sffUPyZfKVh/tDoinkpfLyJJFK+SPmb89cB3JD0FfA1oGcjkMeBuSR8l+QDvih9FRJAklC0RsSQiCsDSou1fLulJkmf4nEQyCmBbZwKPRvK00CbgWyQD7QA0kzwJF5LxBg4C/ybpUpJn5Jgds4rOq5j1OcUPTGsG2ms+ypEM4HJa2wUR8TFJZ5OM7rZI0uuOYJuFNtsvABWSpgGfAs6MiB1ps1JVF9Zb7GCk/QgR0STpLJKniV4GXEfyGHGzY+IrBStLkQxCtFrS+6B1sPRT09czIuKJiLiZZOSzScAeYOgxbHIYySA5uySNJRmHt0Xxun8P/IGk0enY4lcCv2y7svRKZ3hEPAB8nGT8ZrNj5isFK2dXAV+R9FlgAEm/wNPAFyTNJOkjeCgtWwfclDY1/d2Rbiginpa0GHiOZKjFx4oW3wk8KOmliHhzeqvrI+n2fxIR7Y0hMBT4oaSqtN4njjQms/b4llQzM2vl5iMzM2vl5iPr9yTdQTLeb7EvRcQ3sojHrDdz85GZmbVy85GZmbVyUjAzs1ZOCmZm1spJwczMWv1/RnYoQxWhyAYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult = pd.DataFrame.from_csv('1_nestimators.csv')\n",
    "        \n",
    "# plot\n",
    "test_means = cvresult['test-logloss-mean']\n",
    "test_stds = cvresult['test-logloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-logloss-mean']\n",
    "train_stds = cvresult['train-logloss-std'] \n",
    "\n",
    "x_axis = range(0, cvresult.shape[0])\n",
    "        \n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators4_1.png' )\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#可以看到 当n_estimators=98的时候，模型的性能最佳。n_estimators 最小。learning_rate 就固定为0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise-deprecating',\n",
       "       estimator=XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=0.7,\n",
       "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=1, missing=None, n_estimators=104,\n",
       "       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=3, silent=True,\n",
       "       subsample=0.3),\n",
       "       fit_params=None, iid='warn', n_jobs=-1,\n",
       "       param_grid={'max_depth': range(3, 10, 2), 'min_child_weight': range(1, 6, 2)},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#第一轮参数调整得到调整max_depth，min_child_weight\n",
    "\n",
    "xgb2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "max_depth = range(3,10,2)\n",
    "min_child_weight = range(1,6,2)\n",
    "param_test2 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "\n",
    "gsearch2 = GridSearchCV(xgb1, param_grid = param_test2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2.fit(X_train , y_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'max_depth': 3, 'min_child_weight': 1}, -0.06801869531607153)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " #最佳解\n",
    "    gsearch2.best_params_,     gsearch2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "#调整colsample_bytree & colsample_bylevel\n",
    "xgb3 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'colsample_bytree': 0.8, 'subsample': 0.8}, -0.06761749275633107)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3= dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "\n",
    "gsearch3 = GridSearchCV(xgb3, param_grid = param_test3, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3.fit(X_train , y_train)\n",
    "\n",
    "gsearch3.best_params_,     gsearch3.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/utils/deprecation.py:125: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([2.46249447, 2.95471873, 2.83886251, 2.90953026, 2.7346066 ,\n",
       "        2.68392553, 2.82044148, 3.08288727, 3.06553454, 3.08610935,\n",
       "        2.8983767 , 2.70312562, 2.86884861, 3.2081214 , 3.40017109,\n",
       "        3.39304314, 3.19402528, 2.95634737, 3.056915  , 3.44366112,\n",
       "        3.66780605, 3.55008025, 3.40791469, 3.13770385]),\n",
       " 'std_fit_time': array([0.0839954 , 0.14941179, 0.0257828 , 0.07445279, 0.04732992,\n",
       "        0.06858087, 0.13333779, 0.14571945, 0.03317718, 0.09580084,\n",
       "        0.04552143, 0.06969792, 0.03876894, 0.04013051, 0.04254881,\n",
       "        0.04015146, 0.01925171, 0.02425153, 0.04563065, 0.04004203,\n",
       "        0.02701076, 0.01574571, 0.02667536, 0.10313502]),\n",
       " 'mean_score_time': array([0.06423907, 0.07097092, 0.06182256, 0.06715178, 0.06115432,\n",
       "        0.06765738, 0.06545625, 0.0641489 , 0.06474328, 0.06390529,\n",
       "        0.06314273, 0.05722623, 0.06121583, 0.05884266, 0.05965824,\n",
       "        0.0627171 , 0.05873747, 0.05796132, 0.0627162 , 0.06176438,\n",
       "        0.05920696, 0.05806203, 0.05602551, 0.05531273]),\n",
       " 'std_score_time': array([0.00437222, 0.00568833, 0.00322366, 0.00308733, 0.00456234,\n",
       "        0.00326348, 0.00774139, 0.00459644, 0.00354574, 0.00502918,\n",
       "        0.00999342, 0.00231106, 0.00254984, 0.00289227, 0.00119367,\n",
       "        0.00696449, 0.00151963, 0.00263771, 0.00428614, 0.00836541,\n",
       "        0.00498852, 0.00381154, 0.00286504, 0.00583391]),\n",
       " 'param_colsample_bytree': masked_array(data=[0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.7, 0.7, 0.7, 0.7, 0.7,\n",
       "                    0.7, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.9, 0.9, 0.9, 0.9,\n",
       "                    0.9, 0.9],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_subsample': masked_array(data=[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6, 0.7,\n",
       "                    0.8, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6,\n",
       "                    0.7, 0.8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " 'split0_test_score': array([-0.06804062, -0.06767431, -0.06782692, -0.06748713, -0.06743553,\n",
       "        -0.06746849, -0.06776747, -0.0676423 , -0.06732444, -0.06737219,\n",
       "        -0.0671854 , -0.06715135, -0.06768407, -0.06738645, -0.06730397,\n",
       "        -0.06739096, -0.06728401, -0.0671543 , -0.06760864, -0.06729094,\n",
       "        -0.06725739, -0.0671976 , -0.06733208, -0.0674284 ]),\n",
       " 'split1_test_score': array([-0.069472  , -0.069518  , -0.0693558 , -0.06939073, -0.06923596,\n",
       "        -0.06921   , -0.06942103, -0.06920682, -0.06921774, -0.06890528,\n",
       "        -0.06908154, -0.06910437, -0.06957413, -0.06955635, -0.06924697,\n",
       "        -0.06912384, -0.06890351, -0.06890547, -0.06956166, -0.06923498,\n",
       "        -0.06926672, -0.069181  , -0.06903483, -0.06906062]),\n",
       " 'split2_test_score': array([-0.06922749, -0.06881099, -0.06882957, -0.06883757, -0.06896591,\n",
       "        -0.0689209 , -0.06908629, -0.06888651, -0.06866687, -0.06890388,\n",
       "        -0.0688532 , -0.0687568 , -0.06919117, -0.06893391, -0.06888736,\n",
       "        -0.06870404, -0.06901632, -0.06876092, -0.06894186, -0.0689001 ,\n",
       "        -0.06867854, -0.06877879, -0.06883778, -0.06867792]),\n",
       " 'split3_test_score': array([-0.06737058, -0.06725715, -0.0671079 , -0.066944  , -0.06691774,\n",
       "        -0.06703894, -0.06736122, -0.06702005, -0.06704839, -0.06694659,\n",
       "        -0.06691171, -0.06683815, -0.06700473, -0.06683766, -0.06685646,\n",
       "        -0.06679344, -0.06690055, -0.06684489, -0.06706941, -0.06701945,\n",
       "        -0.06654584, -0.06676335, -0.06657861, -0.06667768]),\n",
       " 'split4_test_score': array([-0.06681002, -0.06697891, -0.06668693, -0.06682224, -0.06661474,\n",
       "        -0.06670266, -0.06667948, -0.0667188 , -0.06654875, -0.06647002,\n",
       "        -0.06661334, -0.06656876, -0.0668773 , -0.0664657 , -0.06668667,\n",
       "        -0.06620986, -0.0662976 , -0.06642177, -0.06666198, -0.06664114,\n",
       "        -0.06667941, -0.0664367 , -0.0663886 , -0.06641319]),\n",
       " 'mean_test_score': array([-0.06818417, -0.06804789, -0.06796145, -0.06789636, -0.067834  ,\n",
       "        -0.06786822, -0.06806312, -0.06789492, -0.06776126, -0.06771961,\n",
       "        -0.06772906, -0.06768391, -0.06806631, -0.06783604, -0.06779631,\n",
       "        -0.06764446, -0.06768042, -0.06761749, -0.06796874, -0.06781734,\n",
       "        -0.06768561, -0.06767151, -0.06763441, -0.06765159]),\n",
       " 'std_test_score': array([0.00103129, 0.00096448, 0.00100687, 0.00103414, 0.00107066,\n",
       "        0.00101139, 0.00103782, 0.00099165, 0.00101101, 0.00100877,\n",
       "        0.00102969, 0.00104032, 0.00111585, 0.00120345, 0.00106319,\n",
       "        0.00110976, 0.00109161, 0.00102054, 0.00110729, 0.00104683,\n",
       "        0.00109368, 0.00110261, 0.00111062, 0.00105553]),\n",
       " 'rank_test_score': array([24, 21, 19, 18, 14, 16, 22, 17, 11,  9, 10,  7, 23, 15, 12,  3,  6,\n",
       "         1, 20, 13,  8,  5,  2,  4], dtype=int32),\n",
       " 'split0_train_score': array([-0.06604457, -0.06586715, -0.06589933, -0.06589063, -0.06597154,\n",
       "        -0.06598394, -0.06607953, -0.06598742, -0.06599322, -0.06582628,\n",
       "        -0.06587688, -0.0657979 , -0.06582701, -0.06568877, -0.06560969,\n",
       "        -0.06569049, -0.06574369, -0.06576617, -0.06588813, -0.06556634,\n",
       "        -0.06566462, -0.06573193, -0.06566933, -0.0657238 ]),\n",
       " 'split1_train_score': array([-0.06567831, -0.06543768, -0.06528031, -0.06537901, -0.0653621 ,\n",
       "        -0.06534684, -0.06550137, -0.06533448, -0.06536751, -0.06544836,\n",
       "        -0.06551462, -0.06547556, -0.06525263, -0.06528965, -0.06525604,\n",
       "        -0.06514306, -0.06522298, -0.06530256, -0.0653135 , -0.06511643,\n",
       "        -0.06506507, -0.06517875, -0.06512692, -0.06532702]),\n",
       " 'split2_train_score': array([-0.06578964, -0.06557169, -0.06546038, -0.06543933, -0.06552602,\n",
       "        -0.06556252, -0.06539464, -0.06552619, -0.06539239, -0.06544525,\n",
       "        -0.06539026, -0.06543757, -0.06546399, -0.06523538, -0.0653337 ,\n",
       "        -0.06519613, -0.06546255, -0.06539357, -0.06519832, -0.06521239,\n",
       "        -0.06527482, -0.06524348, -0.06539684, -0.06531991]),\n",
       " 'split3_train_score': array([-0.06634354, -0.06636193, -0.06593425, -0.06586663, -0.06603471,\n",
       "        -0.06610962, -0.06598911, -0.06594903, -0.06591446, -0.06597497,\n",
       "        -0.06596538, -0.06589795, -0.06601808, -0.06599504, -0.06599965,\n",
       "        -0.06585288, -0.06585959, -0.06592269, -0.0657922 , -0.06560085,\n",
       "        -0.06555271, -0.06556395, -0.06560177, -0.06567045]),\n",
       " 'split4_train_score': array([-0.06613904, -0.06615519, -0.06626056, -0.06597755, -0.06592589,\n",
       "        -0.06605855, -0.06618595, -0.06608716, -0.06610251, -0.06604617,\n",
       "        -0.06611641, -0.06596549, -0.0659855 , -0.06601232, -0.06611197,\n",
       "        -0.06602438, -0.06598391, -0.06601255, -0.06603388, -0.06593153,\n",
       "        -0.06591855, -0.0658233 , -0.0657202 , -0.06587839]),\n",
       " 'mean_train_score': array([-0.06599902, -0.06587873, -0.06576697, -0.06571063, -0.06576405,\n",
       "        -0.06581229, -0.06583012, -0.06577686, -0.06575402, -0.06574821,\n",
       "        -0.06577271, -0.06571489, -0.06570944, -0.06564423, -0.06566221,\n",
       "        -0.06558139, -0.06565454, -0.06567951, -0.06564521, -0.06548551,\n",
       "        -0.06549515, -0.06550828, -0.06550301, -0.06558391]),\n",
       " 'std_train_score': array([0.00023961, 0.00034608, 0.00035213, 0.00024962, 0.0002686 ,\n",
       "        0.0003025 , 0.00031994, 0.00029285, 0.00031131, 0.00025612,\n",
       "        0.00027531, 0.00021789, 0.0003014 , 0.00033269, 0.00034404,\n",
       "        0.00035282, 0.00027625, 0.00028334, 0.00032906, 0.00029312,\n",
       "        0.00029827, 0.00025732, 0.00021786, 0.00022337])}"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.067617 using {'colsample_bytree': 0.8, 'subsample': 0.8}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzsnXd4VMX3h99JISEJSUgDUuggBNIIvSPSpHdQmohSREX5UmyAHRFRQCn+UFFAkSIdKdKblFRCUCDUFJKQQEIICUl2fn/sEgOEkLLLpsz7PD7svXdm7rnrZj87Z86cI6SUKBQKhUKhb0yMbYBCoVAoSidKYBQKhUJhEJTAKBQKhcIgKIFRKBQKhUFQAqNQKBQKg6AERqFQKBQGQQmMQqFQKAyCEhiFQqFQGAQlMAqFQqEwCGbGNsCYODk5yerVqxvbDIVCoShRBAQE3JBSOj+pnUEFRgjRFZgPmALLpJSzH7puAfwC+AMJwGAp5WXdNW9gKWALaIAmUso0IcRQ4F1AAtHAMCnlDSHEl0BP4B4QAbwkpbyVl33Vq1fn1KlT+npchUKhKBMIIa7kp53BXGRCCFPgO6Ab4AkMFUJ4PtTsZeCmlLI28DXwha6vGbASGCelbAC0BzJ05+cDHaSU3kAoMFE31m6goe78OeAdQz2bQqFQKJ6MIddgmgIXpJQXpZT3gNVA74fa9AZ+1r1eB3QUQgigMxAqpQwBkFImSCmzAKH7z1rXzhbtLAYp5S4pZaZurL8Bd8M9mkKhUCiehCEFxg24luM4Uncu1zY6cUgCHIG6gBRC7BRCBAohpuraZADjgdNohcUT+CGXe48G/tTfoygUCoWioBTXRX4zoDXQBEgF9gghAoCDaAXGD7gILETrCvvkfkchxHtAJrAqt4GFEK8CrwJUrVrVcE+gUBSAjIwMIiMjSUtLM7YpCkU2lpaWuLu7Y25uXqj+hhSYKMAjx7G77lxubSJ16yt2aBf7I4GDUsobAEKI7UAjIBlAShmhO78GmH5/MCHEKKAH0FE+ptCNlPJ74HuAxo0bq2I4imJBZGQkFSpUoHr16mi9vwqFcZFSkpCQQGRkJDVq1CjUGIZ0kZ0E6gghagghygFDgM0PtdkMjNS9HgDs1QnDTsBLCGGlE552QDhaQfIUQtwPj+sEnIXsiLWpQC8pZaoBn0uh0DtpaWk4OjoqcVEUG4QQODo6FmlWbbAZjJQyUwgxEa1YmAI/SinPCCE+Ak5JKTejXT9ZIYS4ACSiFSGklDeFEPPQipQEtksptwEIIT4EDgohMoArwCjdLb8FLIDduj/Sv6WU4wz1fAqFvlHioihuFPUzadA1GCnldmD7Q+dm5HidBgx8TN+VaEOVHz6/BFiSy/naRbU3v1y6cYd1Add467m6mJmqZAgKhUKRG+rbsRDsOnOd7/ZF8OKy48TfTje2OQqFQlEsUQJTCMa2q8W8QT6ERN6ix8JDBFxJNLZJCkWRuHXrFosWLSpU32+++YbUVP0tey5fvpyJEyc+uaGe2b9/Pz169Chwv4Lae/nyZX799dcC3ye/JCYm0qlTJ+rUqUOnTp24efNmru2uXr1K586dqV+/Pp6enly+fFnvtiiBKST9GrmzYUIrLM1NGbz0b34+epnHBK4pFMWe4iQwpZ28BCYzMzPX8wVh9uzZdOzYkfPnz9OxY0dmz56da7sRI0YwZcoUzp49y4kTJ3BxcSnyvR+muO6DKRHUr2LL5omtmbwmmJmbzxB09Saf9fPCqpx6WxWF58MtZwiPTtbrmJ6utszs2eCx16dPn05ERAS+vr506tQJFxcX1qxZQ3p6On379uXDDz/kzp07DBo0iMjISLKysvjggw+IjY0lOjqaDh064OTkxL59+3Idf8eOHbz77rtkZWXh5OTEnj17SExMZPTo0Vy8eBErKyu+//57vL29H+i3du1aPvzwQ0xNTbGzs+PgwYNcvnyZ4cOHc+fOHQC+/fZbWrZsyf79+5k5cyb29vacPn2aQYMG4eXlxfz587l79y4bN26kVq1ajBo1CktLS06dOkVycjLz5s17ZOZy584dXn/9dcLCwsjIyGDWrFn07v1wIpL/uHbtGu3btycqKophw4Yxc+ZMZsyYgYODA5MmTQLgvffew8XFhd9++42zZ8/i6+vLyJEjqVixIn/88QcpKSlkZWVx4MABvvzyy0fef4CVK1eyYMEC7t27R7NmzVi0aBGmpqYP2LJp0yb2798PwMiRI2nfvj1ffPHFA23Cw8PJzMykU6dOANjY2Dz22YqC+iYsInblzfl+eGMW7b/AV7vPcTbmNkuG+1PDydrYpikU+Wb27NmEhYURHBzMrl27WLduHSdOnEBKSa9evTh48CDx8fG4urqybds2AJKSkrCzs2PevHns27cPJyenXMeOj4/nlVde4eDBg9SoUYPERK1LeebMmfj5+bFx40b27t3LiBEjCA4OfqDvRx99xM6dO3Fzc+PWLW3uWhcXF3bv3o2lpSXnz59n6NCh2UlrQ0JCOHv2LA4ODtSsWZMxY8Zw4sQJ5s+fz8KFC/nmm28A7SzixIkTRERE0KFDBy5cuPDAfT/99FOeffZZfvzxR27dukXTpk157rnnsLbO/e/6xIkThIWFYWVlRZMmTejevTujR4+mX79+TJo0CY1Gw+rVqzlx4gQ+Pj7MnTuXrVu3AloXW2BgIKGhoTg4OLBr1y7Onz//yPvv7OzM77//zpEjRzA3N2fChAmsWrWKESNGMGbMGMaNG0fjxo2JjY2lSpUqAFSuXJnY2NhH7D137hz29vb069ePS5cu8dxzzzF79uxHxKqoKIHRAyYmgonP1sHb3Z43VgfRa+FhvhrkQ+cGlY1tmqIEktdM42mwa9cudu3ahZ+fHwApKSmcP3+eNm3aMHnyZKZNm0aPHj1o06ZNvsb7+++/adu2bfZmPQcHBwAOHz7M+vXrAXj22WdJSEggOfnBmVurVq0YNWoUgwYNol+/foA268HEiRMJDg7G1NSUc+fOZbdv0qRJ9pdrrVq16Ny5MwBeXl4PzK4GDRqEiYkJderUoWbNmvzzzz+PvAebN29m7ty5gHaf0tWrV6lfv36uz9ipUyccHR0B6NevH4cPH2bSpEk4OjoSFBREbGwsfn5+2W1y63//fXnc+x8aGkpAQABNmjQB4O7du9lurWXLluU6rhAi11DjzMxMDh06RFBQEFWrVmXw4MEsX76cl19+OddxCosSGD3Stq4zWya2ZsKqQF5dEcBrHWrxdqdnMDVR+xsUJQcpJe+88w5jx4595FpgYCDbt2/n/fffp2PHjsyYMSOXEfTHkiVLOH78ONu2bcPf35+AgAAWLlxIpUqVCAkJQaPRYGlpmd3ewsIi+7WJiUn2sYmJyQPrGw9/6T58LKVk/fr1PPPMM/my83HjjRkzhuXLl3P9+nVGjx792P45Z0aPe/8XLlzIyJEj+fzzz/O0pVKlSsTExFClShViYmJyXVtxd3fH19eXmjVrAtCnTx/+/vtvvQuMWuTXMx4OVqwd14IhTTz4bl8Eo346QeKde8Y2S6HIkwoVKnD79m0AunTpwo8//khKSgoAUVFRxMXFER0djZWVFcOGDWPKlCkEBgY+0jc3mjdvzsGDB7l06RJAtousTZs2rFqlTRm4f/9+nJycsLW1faBvREQEzZo146OPPsLZ2Zlr166RlJRElSpVMDExYcWKFWRlZRX4edeuXYtGoyEiIoKLFy8+IiRdunRh4cKF2YE7QUFBeY63e/duEhMTs9d6WrVqBUDfvn3ZsWMHJ0+epEuXLsCT36/Hvf8dO3Zk3bp1xMXFAdr38cqVR8uy9OrVi59/1iap//nnn3NdO2rSpAm3bt0iPj4egL179+Lp+XA1laKjZjAGwNLclNn9vfH1sGfG5jP0XHiYRS82wsfD3timKRS54ujoSKtWrWjYsCHdunXjhRdeoEWLFoB2AXjlypVcuHCBKVOmYGJigrm5OYsXLwbg1VdfpWvXrri6uua6yO/s7Mz3339Pv3790Gg02Wsos2bNYvTo0Xh7e2NlZZX9pZiTKVOmcP78eaSUdOzYER8fHyZMmED//v355Zdf6Nq162PXRfKiatWqNG3alOTkZJYsWfLALAjggw8+YNKkSXh7e6PRaKhRo0b2mkluNG3alP79+xMZGcmwYcNo3LgxAOXKlaNDhw7Y29tnr294e3tjamqKj48Po0aNomLFig+M1blzZ86ePfvI++/p6cknn3xC586d0Wg0mJub891331GtWrUH1mCmT5/OoEGD+OGHH6hWrRpr1qwB4NSpUyxZsoRly5ZhamrK3Llz6dixI1JK/P39eeWVVwr8Pj4JUZZDaxs3biwNXdHydGQS41YGEH87nQ97N2BIEw+VEkTxCGfPnn2sf1+hX0aNGkWPHj0YMGCAwe+l0Who1KgRa9eupU6dOga/nyHI7bMphAiQUjZ+Ul/lIjMwXu52bH29Nc1rOfLOH6eZtj6UtIyCT+kVCkXJIjw8nNq1a9OxY8cSKy5FRbnIngIVrcvx06gmzP/rHAv2XiA8JpnFL/rj4WBlbNMUCr3SrFkz0tMfTJ+0YsUKvLy8jGTRoyxfvrxQ/Xbu3Mm0adMeOFejRg02bNiQa3tPT08uXrxYqHuVFpSLzMAusof5KzyWt9YEY2oi+GawL+2f0f/uWUXJQ7nIFMUV5SIrQTznWYktE1tT2daSl5afZMGe82g0ZVfkFQpF6UUJjBGo7mTNhgmt6OPrxrzd5xjzyymSUjOMbZZCoVDoFSUwRqJ8OVPmDfLh494NOHQ+np7fHuZMdJKxzVIoFAq9oQTGiAghGN6iOqtfbUF6Zhb9Fh1lfUCksc1SKBQKvaAEphjgX60iW19vg19VeyavDeH9jadJz1ShzIqnR3FK16/qwRSN/NSD2bdvH76+vtn/WVpasnHjRr3bogSmmOBcwYKVLzdjbNuarPz7KoOX/k1M0l1jm6UoIxQngSntFId6MB06dCA4OJjg4GD27t2LlZVVdmJQfaL2wRQjzExNeOf5+vh62PO/tSH0WHCYhUP9aFk79zToilLKn9Ph+mn9jlnZC7rlXngKVD2YslYPJifr1q2jW7duWFnpf1+eEphiSDevKtSpVIFxKwMY9sNxpnatx9i2NVWKGYXBUPVgylY9mJysXr2at99+O882hUUJTDGltosNm15rxdT1ocz+8x+Crt5k7kAfKliaG9s0haHJY6bxNFD1YEp/PZj7xMTEcPr06exMz/pGCUwxxtrCjG+H+uHnYc/nf/5D72+PsGS4P3UrVTC2aYpSjKoHU/rrwdxnzZo19O3bF3Nzw/xwVYv8xRwhBGPa1OTXMc1ITsuk97dH2BISbWyzFKUMVQ+mbNWDuc9vv/3G0KFD83y2oqBmMCWEZjUd2faGtlrm678FEXT1Fu88Xw9zU/UbQVF0VD2YslUPBrTrUNeuXaNdu3YFfv/yi0p2+ZSTXRaVe5kaPtt+luVHL9OkekW+e6ERLraWT+6oKNaoZJdPD1UPpmAU22SXQoiuQoh/hRAXhBDTc7luIYT4XXf9uBCieo5r3kKIY0KIM0KI00IIS935obrjUCHEDiGEk+78QF1bjRDiiQ9eUilnZsKsXg2YP8SXsKhkui88zMnLicY2S6FQPISqB2PAGYwQwhQ4B3QCIoGTwFApZXiONhMAbynlOCHEEKCvlHKwEMIMCASGSylDhBCOwC1AANGAp5TyhhBiDpAqpZwlhKgPaIClwP+klE+cmpTEGUxO/rmezLgVAUTevMu7z9fnpVbVVShzCaW0zGBKQj2YwlLQejClhaLMYAy5BtMUuCClvKgzaDXQGwjP0aY3MEv3eh3wrdB+Q3YGQqWUIQBSygTdGOZoRcZaCJEA2AIXdG3O6toY8JGKF/Uq27L59dZMXhPCR1vDCbx6ky/6e2NtoZbWFMbh+PHjxjbBYHTp0sVg4bylFUO6yNyAazmOI3Xncm0jpcwEkgBHoC4ghRA7hRCBQoipujYZwHjgNLqZDPCDAZ+h2GNrac7SYf5M6fIM20/H0HfRES7GpxjbLIVCoSi2YcpmQGvgRd2/fYUQHXUzmPGAH+AKhALvFGRgIcSrQohTQohT8fHxejbbOJiYCF7rUJtfRjfjRso9en17hB1h141tlkKhKOMYUmCiAI8cx+66c7m20a272AEJaGc7B6WUN6SUqcB2oBHgCyCljJDaxaM1QMuCGCWl/F5K2VhK2djZ2bngT1WMaV3HiS2vt6aWszXjVgYw+89/yMzSGNsshUJRRjGkwJwE6gghagghygFDgM0PtdkMjNS9HgDs1QnHTsBLCGGlE552aNduogBPIcR9ZegEnDXgM5Q43OzLs2ZcC15oVpUlByIY8eMJbqSkP7mjokxTnLIpq3T9RSM/6foBpk6dSoMGDahfvz5vvPEGhgj4MpjA6NZUJqIVi7PAGinlGSHER0KIXrpmPwCOQogLwNvAdF3fm8A8tCIVDARKKbdJKaOBD4GDQohQtDOazwCEEH2FEJFAC2CbEGKnoZ6tuGNhZspnfb2YM8CbU1du0nPhYYKu5v4hUyigeAlMaac4pOs/evQoR44cITQ0lLCwME6ePMmBAweKfO+HMegajJRyu5SyrpSylpTyU925GVLKzbrXaVLKgVLK2lLKpvcjznTXVkopG0gpG0opp+Y4v0RKWV9K6S2l7Hk/wkxKuUFK6S6ltJBSVpJSlvlwj0GNPfhjfEtMTQSDlh5j5d9XDPIrRVHyyZmuf8qUKXz55Zc0adIEb29vZs6cCWhT2Hfv3h0fHx8aNmzI77//zoIFC7LT9Xfo0OGx4+/YsYNGjRrh4+NDx44dAe0v7T59+uDt7U3z5s0JDQ19pN/atWtp2LAhPj4+tG3bFtB+Qbdp04ZGjRrRqFEjjh49CmhnIO3ataN3797UrFmT6dOns2rVKpo2bYqXlxcRERGAdqPl/V3vdevWzXWH/p07dxg9ejRNmzbFz8+PTZs25fn+3U/XX6dOnezU+jNmzMjO3gzadP3z589n+vTpHDp0CF9fX77++muWL19Or169ePbZZ7Pfm9zef9Cm62/atCm+vr6MHTs21zQ5mzZtYuRIrWNo5MiRuRYSE0KQlpbGvXv3SE9PJyMjg0qVKuX5jIVBxbOWchq62bH19da8uTqY9zeGEXT1Fp/2bYiluemTOyuMwhcnvuCfxH+e3LAA1HOox7Sm0x57XaXrL1vp+lu0aEGHDh2oUqUKUkomTpxokH1YSmDKAPZW5fhpVBPm7znPgr3nORuTzJJh/lR11H+BIUXJR6XrL/3p+i9cuMDZs2eJjIzMvv+hQ4fy/f80vyiBKSOYmAje6lQXXw973lwdRI+Fh5g/xI8O9R6fylthHPKaaTwNVLr+0p+uf8OGDTRv3hwbGxsAunXrxrFjx/QuMMV1H4zCQHSo58LW19vgXtGKl5afZN7uc2Rp1LpMWUel6y9b6fqrVq3KgQMHyMzMJCMjgwMHDigXmUI/VHW0Yv34lry/MYwFe84Tcu0W84f4Ym9VztimKYyEStdfttL1DxgwgL179+Ll5YUQgq5du9KzZ88Cv49PQqXrL0Syyxt3b7D7ym6G1jNcoZ6ngZSSX09cZdbmM1SytWTJMH8autkZ26wySWlJdlkSUOn6C0axTddfWlkbvorPjn/Gn5f+NLYpRUIIwYvNqrFmbAuyNJL+i4+y5tS1J3dUKBRPRKXrVy6yQjHm5k2OpaUx6/B71HOoRw27GsY2qUj4Va3I1tdb8/pvQUxdF0rQ1VvM6uWJhZkKZVYUjJKQrn/58uWF6lfQdP2enp5cvHgx12tlBeUiK0w9mMx0rq95kUFp4TiVd2bVwB2UNyuvfwOfMplZGr7afY7F+yPwcbdj0TB/3OxL/nOVBJSLTFFcUS6yp42ZBZUH/8bs8s9wIT2Bzze9YGyL9IKZqQnTutZjyTB/IuLv0GPBIQ6fv2FssxQKRQlFCUxhMTWn5aC1vGrhwYaUC2zcPBpKyWywa8PKbJ7YCucKFoz48Tjf7buARoUyKxSKAqIEpiiYmjF+4GaamtnzacJxzm+bWGpEpqazDRsmtKK7tytf7vyXsSsDSE7LMLZZCoWiBKEEpoiYmpnzRd8/sDErz9sxe0jd+gZoSkcNFmsLMxYM8WVmT0/2/RNH168P8uvxq9zLLB3Pp1AoDIsSGD3gZOXMnOcWcbWcOR9e3Ybc9BpoCr67uDgihOClVjX4fWxznG0teXfDadp/uY8Vf18hPbN0PKOieKXrV/VgikZ+68FMmzaNhg0bZmfGNgRKYPREkypNmej3OtttrFl7cTOsHwNZpcel5F/NgY0TWrL8pSZUsrPkg41htP9yP78cu0xahhKakk5xEpjSTnGoB7Nt2zYCAwMJDg7m+PHjzJ0795FEo/pACYweedlrDK3cWjHbyYnw81tg7SjILD3VJIUQtH/GhT/Gt2TFy01xsy/PjE1naPflPpYfuaSEpgSj6sE8SGmvBxMeHk7btm0xMzPD2toab29vduzYkeczFga10VKPmAgTPm/9OQO3DGRydRvW/LuNCqtfhMErwLz07CcRQtCmjjOtaztxNCKB+X+dZ9aWcBbtj2Bsu1q82KyqqjdTBK5/9hnpZ/VbD8aifj0qv/vuY6+rejBlqx6Mj48PH374IZMnTyY1NZV9+/bh6en52M9HYVECo2cqWlZkbru5vLTjJWZ4P8u8kL8QqwbC0NVgYWNs8/SKEIJWtZ1oVduJYxEJzN9zjo+3hrN4fwTj2tXkxWbVKF9OCU1JQ9WDKf31YDp37szJkydp2bIlzs7OtGjRIjsZpz5RAmMAfF18meQ/ibmn5rKq9csMO/IjrOwPL64By9KZTLJFLUda1GrB8YsJzN9znk+2nWXJgQhebVuTYc2rYVVOfdTyS14zjaeBqgdT+uvBgNZl99577wHwwgsvULdu3TzHLQxqDcZAjPAcQQePDnwVvZeQrh9B1Cn4pQ+kJhrbNIPSrKYjv77SnLXjWlCvsi2fbf+H1l/sY/H+CO6kF30BU2EYVD2YslUPJisri4SEBABCQ0MJDQ3Nnu3pE/Wz0kAIIfi41ccM3jqYKde2sKbfUuw3jIefe8GIjWCdu7+6tNCkugMrxzQj4Eoi8/dc4Isd//D9wQjGtKnJyJbVsbFQH73ihKoHU7bqwWRkZGS7OG1tbVm5ciVmZvr/m1TJLguT7LIAnLlxhuF/Dqd5leZ8W60vJr8PB/uqMHIzVKhs0HsXJwKv3mTBnvPs/zceeytzXm5Vg1GtqlPB0tzYphULVLLLp4eqB1MwVLLLYkwDpwZMbTKVQ1GH+CntKgxbB0mR8FM3uFV2aq80qlqR5S81ZdNrrfCvWpGvdp+j1ey9zP/rPEl3S89+IYXiPqoejJrBGHwGA9pFu6kHp7L7ym6WdV5G4wwNrBygXfAfuQkcahrchuLG6cgk5u85z19nY6lgacboVjUY3aoGdlZlc0ZTWmYwJaEeTGEpaD2Y0kJRZjBKYJ6CwADcybjDkK1DuJNxh7U91+J48xqs6ANmljBiMzjrP4KjJBAWlcSCPefZFR5LBQszRrWqzsuta2BvVc7Ypj1VSovAKEofykVWArA2t2Zuu7kk30tm+qHpZFX2glHbQJMJy5+H2DPGNtEoNHSz4/sRjdn+Rhta13Fi4d4LtP5iH1/u/Iebd+4Z27ynSln+saconhT1M2lQgRFCdBVC/CuEuCCEmJ7LdQshxO+668eFENVzXPMWQhwTQpwRQpwWQljqzg/VHYcKIXYIIZx05x2EELuFEOd1/1Z8+H7G5hmHZ3iv2Xv8HfM334d+D5UawKjtYGIGy7tDdPCTBymleLrasniYPzsmtaFdXWcW7Y+g9Rd7+WLHPySWAaGxtLQkISFBiYyi2CClJCEh4ZEIu4JgMBeZEMIUOAd0AiKBk8BQKWV4jjYTAG8p5TghxBCgr5RysBDCDAgEhkspQ4QQjsAtQADRgKeU8oYQYg6QKqWcpXudKKWcrROzilLKBx2mD/E0XWT3kVLy/pH32RKxhaWdltLCtQUkXoSfe0NakjYIwKPpU7WpOHIu9jYL915ga2g05c1NGd68Gq+0rYmTjcWTO5dAMjIyiIyMJC0tzdimKBTZWFpa4u7ujrn5g2ujRl+DEUK0AGZJKbvojt8BkFJ+nqPNTl2bYzpRuQ44A92AF6SUwx4a0xytwDQGrgKLgUAp5fdCiH+B9lLKGCFEFWC/lDLPbbjGEBiA1IxUXtz+IolpiaztuRYXKxdtRNkvvSAlDl74Haq3fup2FUcuxGmFZktINBZmpgxrXpVX29bCuULpFBqFoiRQHNZg3ICccbiRunO5tpFSZgJJgCNQF5BCiJ1CiEAhxFRdmwxgPHAa3UwG+EE3ViUpZYzu9XWgUm5GCSFeFUKcEkKcio+PL+IjFg4rcyu+avcVdzPvMuXAFDI1mWDvAS/9CbZu2gizC3uMYltxo7ZLBeYP8WP32+3o1rAyPxy+RJs5e/l4azhxt9WvfYWiOFNcF/nNgNbAi7p/+wohOupmMOMBP8AVCAXeebiz1E7Lcp2aSSm/l1I2llI2dnZ2NpT9T6SmfU1mtphJYFwg3wZ9qz1ZobJ24d+xNvw2BP7902j2FTdqOdswb7Aveya3p7uXK8uPXqbNF/v4cMsZYpOV0CgUxRFDCkwU4JHj2F13Ltc2OheZHZCAdrZzUEp5Q0qZCmwHGgG+AFLKCJ2IrAFa6saK1bnG0P0bZ4iH0ifda3ZnYN2B/BD2AwcjD2pP2jhrd/lXagi/D4MzpTvGvqDUcLLmq0E+7Hm7Hb18XPnl2BXazNnHzE1hxCTdNbZ5CoUiB4YUmJNAHSFEDSFEOWAIsPmhNpuBkbrXA4C9OuHYCXgJIax0wtMOCEcrSJ5CiPtTj07A2VzGGgnkXSGomDCt6TTqOdTjnUPvEJ0SrT1p5aDNV+bWGNaNhhDDlDMtyVR3subLgT7sm9yefn5urDp+lXZz9vPBxjCibymhUSiKAwbdaCmEeB74BjAFfpRSfiqE+Ag4JaXcrAs9XoHW5ZUIDJFSXtT1HYbW/SWB7VLKqbrz44A3gQz+ssLjAAAgAElEQVTgCjBKSpmgizRbA1TVnR8kpcwzdbGxFvkf5mryVQZvHUxNu5os77occ1NdxEZ6itZVdvkw9JwP/iPzHqgMcy0xlUX7I1gXoF32G9jYgwnta+Fe0crIlikUpQ+jR5GVBIqLwADsvrKbt/e/zbD6w5jWNEd0dcZdravswl/QbQ40e7RGh+I/om7dZfH+C6w5GYlEMsDfnQnta+PhoIRGodAXxSGKTFEAOlXrxLD6w1h5diV/Xfnrvwvm5WHIr1CvB/w5FQ5/8/hBFLjZl+eTPl7sn9KeoU2rsj4gig5z9zNtXShXE1KNbZ5CUaZQM5hiMoMByMjKYOSOkVxKusSaHmvwsM0RI5GVARvGQth6aP8OtJsGuZRCVTzI9aQ0lhyI4NcTV8nSSPr6uTGxQ22qOxW8hohCodCiXGT5oLgJDEB0SjQDtwzEzcaNFc+vwMI0x4ZCTRZsfh2CV0GrSfDcLCUy+SQuOY0lBy6y6vgVMjWS3r6uvP5sHWoooVEoCoxykZVQXG1c+az1Z5xNPMucE3MevGhiCr2+hcaj4cg3sGM6lOEfCAXBxdaSGT09OTStAy+1rM720zF0/Go/b/0ezIW4FGObp1CUSpTAFILMmze5tWGjwcZv59GO0Q1Hs+bcGrZd3PbgRRMT6D4Pmk+A40tg6yTQaAxmS2nDpYIl7/fw5NDUZxnTpiY7wq7T6esDvPFbEBfiHl8nXaFQFBzlIiuEiyxu/nwSFi/BYeRIXKZOQehqbeuTTE0mL+98mbOJZ1ndYzU17R4qSiYl7P0YDn0F3kOg93dgqurcF5SElHT+79Alfjl2mbsZWXT3qsIbHetQt1IFY5umUBRb1BpMPiiswMisLGI/n83NlSup0Ok5XOfMwaR8eb3bF3snlkFbB+Fg6cCv3X+lvFku9zjwJez7BDz7QP9lYFo2K0IWlcQ791h26CI/H73MnXtZPO9VmTc61qFeZVtjm6ZQFDuUwOSDoi7yJ/78M7Gzv8DSywuPxYswc3TUo3VajkYfZdzucfSq1YtPWn/ymEYLYdf78MzzMHA5mKlMw4XlVuo9fjh8ieVHLnM7PZNn67kwqLE7Heq5YGGm/5mqQlESUQKTD/QRRZa8ezfRU6Zi5uSEx/dLsahZ88mdCsii4EUsDlnMRy0/om+dvrk3OvF/sP1/UOtZGLwKyqmNhUUhKTWDH49cYvXJq8Qmp2NvZU5vH1f6+7vj5WaHUNF7ijKM3gRGCFELiJRSpgsh2gPewC9Sylt6sdSI6CtM+W5oKNfGT0BmZuLx7UKsmjTRg3X/kaXJYuxfYwmOC2bV86t4xuExZW4CV2jDmKu3hqGrwcJGr3aURbI0ksMXbrAuIJJdZ66TnqmhbiUb+jdyp6+fGy62ha/2p1CUVPQpMMFoC3xVR5vVeBPQQEr5vB7sNCr63AdzLzKSa6+OJePaNap89hl2PXvoZdz73Lh7g0FbBmFtbs3qHquxNn/M/o3QtdoNmW7+2uqYlnZ6taMsk3Q3g22hMawLuEbg1VuYCGhb15n+jdzp5FkJS3PlQlOUDfQpMIFSykZCiClAmpRyoRAiSErppy9jjYW+N1pmJSUROfF1Uk+exHnSmziOHatXV8qp66d4edfLdKnWhS/afvH4scM3wbqXoVIDGL5Bm51ZoVcuxqfwR2AUfwRGEp2Uhq2lGT11LjQ/D3vlQlOUavQpMMfRZkR+D+gppbwkhAiTUjbUj6nGwxA7+TX37hHz/vskb96CXf9+VJk1C2Guv8iuZaeXMT9wPu83e5/B9QY/vuG5nfD7cG3xshEbwcZFbzYo/kOjkRy7mMC6gEj+DIshLUNDTWdr+jdyp18jN6rY6T+6UKEwNvoUGE9gHHBMSvmbEKIG2lT4X+jHVONhqFQxUkpuLFzIjUWLsW7ZErf532BaQT/7KjRSw8Q9E/k75m9WPL+CBo4NHt84Yh/8NlRbjnnEJrB11YsNity5nZbBn6evsy4wkhOXEhECWtd2on8jd7o0qEz5csqFpigdGCSKTAhREfCQUoYWxbjigqFzkd1a/wcxM2diUbMmHkuXYF6lin7GTbvFwK0DMRWmrOm5BttyeezVuHIUVg0EaycYuQXsq+rFBkXeXE1IZX1gJOsDI4m8eRcbCzO6e1VhQGN3GlerqFxoihKNPmcw+4FegBkQgLYU8REp5dt6sNOoPI1kl3eOHiXyjTcxKV8ej6VLsPT01Mu4IfEhjPpzFG3d2/JNh2/y/sKKPAUr+4GFrXYm41hLLzYonoxGIzlxOZF1AZFsPx1D6r0sqjlaZbvQVEE0RUlEnwITJKX0E0KMQTt7mSmECJVSeuvLWGPxtLIpp507x7Wx48hKSsL963nYtGunl3FXhK9gzsk5TGk8hRENRuTdOCYEfukDpuVg5GZwfkyos8Jg3EnPZEfYddYHRnI0IgGAFjUd6e/vTreGlbG2UKl+FCUDfQrMaaAz8DPwnpTypBKYgpMRG0fk+PGk/fMPlWd8QMUhQ4o8ppSSt/a/xYFrB/ip60/4uvjm3SHuLPzcC6RGu/Bf2avINigKR+TNVDYERrEuMJIrCalYlTOlW8MqDPB3p1kNB0xMlAtNUXzRp8AMBD5A6xYbL4SoCXwppeyvH1ONx9OuB6O5c4eotyeTcuAADi+PxmXyZIRJ0RJaJ99LZvCWwWRoMljbcy0VLSvm3eHGBfilF9y7ow1hdmtUpPsrioaUkoArN1kXEMm20Bhup2fiXrE8/Rq507+RG9UcVb0aRfFDpYrJB8YoOCYzM4n97DNu/vobFbp2xXX255hYFm03eHhCOMO2D6NZlWZ81/E7TMQTROvmZfi5J9y9BS+uharNi3R/hX64ey+LXeHXWRcQyeELN5ASmlZ3oL+/G897VaGCpUpkqige6HMG4w4sBFrpTh0C3pRSRhbZSiNjrIqWUkoSl/9M3Jw5lPfxwX3Rd5g5FG0z5Jp/1/Dx3x/zZqM3GeM15skdkqK0InP7OrywGmq0LdL9FfolJukuG4KiWBcQycX4O1iam9C1QWUG+HvQopYjpsqFpjAi+hSY3cCvwArdqWHAi1LKTkW20sgYu2Ry8s5dRE+dilmlSngsXYJFjRqFHktKybRD09h5eSfLOi+jSeV85EO7HQu/9Iabl7QJMus8V+j7KwyDlJLga7dYHxjJ5uBoktMyqWJnSb9GbvRv5E5NZ5VvTvH00WsuMiml75POlUSMLTAAd4ODuTZ+Amg0uC/6Dit//0KPdSfjDkO2DiElI4W1PdfiVN4pH50SYEVviP9Xm+q/XvdC319hWNIysvjrbCzrAyI5cC4ejYRGVe3p7+9OD29X7MorF5ri6aBPgdkD/AT8pjs1FHhJStmxyFYameIgMAD3rl7VJsqMisL1i9nYPl/4PKLnbp7jxW0v4uPsw9JOSzE1ycfu8bs3YWV/bShzv++hYYmP3yj1xCWnsTFY60I7F5tCOTMTOntWYoC/O23qOCsXmsKg6FNgqqFdg2kBSOAo8LqU8po+DDUmxUVgADJv3iRy4uvcDQjAefLbOI4ZU+jd3hsvbOSDIx8wzmccr/m+lr9Oacnw6yC4dlxbftn3hULdW/F0kVISFpXMuoBrbAqJ5lZqBi4VLOjbyI0Bjdypo0o/KwyAQaPIhBCTpJTfFMqyYkRxEhgATXo6Me+8S/L27dgPGkTlGR8gzAq3+e6DIx+w6cImljy3hJZuLfPX6d4dbe6ySwegx9fQeHSh7q0wDvcyNez9J451AZHs/zeOTI3Ex92O/v7u9PJxxd6qnLFNVJQSDC0wV6WUT0xqJYToCswHTIFlUsrZD123AH4B/IEEYLCU8rLumjewFLAFNEATwBxtFNt93IGVUspJupnWj4AzkAgMe1KkW3ETGACp0RA/fwEJS5di3aYNbl9/jalNwfdC3M28ywvbXiDhbgJre66lknWl/HXMSIM1I+D8Tug6G5qPL/C9FcbnRko6m4KjWRcQydmYZMqZmtCxvgsD/N1pW9cZc9Oi7b9SlG0MLTDXpJQeT2hjCpwDOgGRwElgqJQyPEebCYC3lHKcEGII0FdKOVgIYQYEAsOllCFCCEfglpQy66F7BABvSSkPCiHWAlullD8LIZ5Fu040PC8bi6PA3Ofm2rVcn/UhFnXq4LFkMeaVKxd4jEtJlxiydQj1HOqxrMsyzE3yuQiceQ/Wj4azW6DjTGhT4tPOlWnORCexPiCKTcFRJNy5h5NNOfr4utHf3536VfJIlKpQPAajz2CEEC2AWVLKLrrjdwCklJ/naLNT1+aYTlSuo52BdANekFIOy2P8usAeoKqUUgohzgBdpZTXhHbxIklKmedfT3EWGICUw0eIevNNTGxstIky69Ur8BjbL25n2qFpvNTwJd72L4BQZGXCxnFwei20mwbt3wGVAbhEk5Gl4cC/8awLiGTPP7FkZEkauNrSv5E7vX1dcbSxMLaJihJCfgXmsfNkIcRtIURyLv/dBvJTWMQNyBkIEKk7l2sbKWUmkAQ4AnUBKYTYKYQIFEJMzWX8IcDv8j+FDAH66V73BSroZj4PP9erQohTQohT8fHx+XgM42HTuhXVfl0FQnDlhRdJOXS4wGM8X/N5Bj8zmJ/CfmL/tf3572hqBn2Xgt8wOPAF7J4BZTjrQ2nA3NSE5zwrsWS4PyfefY4PezXA1ETw0dZwmn22h1d+OcWOsOvcy9QY21RFKcFgqWKEEAPQzijG6I6HA82klBNztAnTtYnUHUcAzYBRwGto111S0c5U3pdS7snRNxytCy1Ad+wKfAvUAA4C/YGGUspbj7OxuM9g7pMRG8u1ceNJP3eOyjNnUHHQoAL1T89KZ/j24USlRLGm5xrcbB7W+TzQaODPKXBymXbRv9scMFX7LUoT52Jvsz4gkj+Cooi/nY69lTndvarQ188Nf1W7RpELRZ7B6IEoIOc6jbvuXK5tdC4yO7SL/ZHAQSnlDSllKrAdyM7KKITwAczuiwuAlDJaStlPSumHtrwzeYlLScK8UiWqrViBdauWXJ8xk7h5XyM1+f+VaWFqwVftv0JKyf/2/4+MrIz839zEBJ6fC63ehFM/wvIekBxdiKdQFFfqVqrAO8/X59j0Z/nppSa0q+vM+sBIBiw5Rps5+5i7818uxN02tpmKEoghBeYkUEcIUUMIUQ6tS2vzQ202AyN1rwcAe3Uur52AlxDCSic87YDwHP2G8t/GTwCEEE5CZGd5fAdtRFmpwdTGGo9Fi7AfPJiE778n+n9T0KSn57u/RwUPPm71MWEJYXwV8FXBbi4EdPoI+v8A10/D0rZwcX/BxlAUe8xMTejwjAvzh/gR8H4nvh7sQ01nGxbtv8Bz8w7SY+Ehlh26SGxymrFNVZQQDJpNWQjxPPAN2jDlH6WUnwohPgJOSSk3CyEs0eY480MbWjxESnlR13cYWqGQwHYp5dQc414EnpdS/pPj3ADgc137g8BrUso8v4FLiossJ1JKEn/4gbi5X1He3x/3bxdiVvEJKfpzMOfkHFaEr+Crdl/RuXrnghsQ/y/8PhwSzkP7d6HNZO0sR1FqibudxtaQGDYFRxESmYSJgJa1nOjj50aXBpVUlucyiErXnw9KosDcJ/nPP4meNh3zKlXw+H4p5apVy1e/jKwMRu0cRcStCNb0WENV2yduZ3qU9BTY+hacXgO1O2nTy1gVLRu0omQQEZ/CpuBoNgZFcTUxFQszbeBAX1832tZ1ppyZ+rFRFtBnqpjbaGcFOUkCTgGT7884SiIlWWAAUgMDiZzwGgihTZTp55evfjEpMQzcOpAq1lVY0W0FlmaFqEcjpXZNZsd0sKkEA38G98In6lSULKSUBF27xcagKLaGxpB45x72Vub08K5CH18VHFDa0afAfIx20f1XQKBdS6mFdiPkeCll+yJbayRKusAA3Lt8matjx5IZcx3XOXOw7dolX/0ORh7ktT2vMaDuAGa2mFl4A6ICYc1IuB0DXT+HJmPUfpkyRkaWhkPn49kYFM2u8OukZWjwcChPbx83+vi5UttF5UMrbehTYEKklD4PnQuWUvrmdq0kURoEBnSJMl+byN3AQFymTMFh9Ev5+vU4P3A+y04v47PWn9GzVs/CG5CaCBvGadPLNOwPPReAhapTUhZJSc9k15nrbAiK4siFG2gkNHSzpY+vG718XHGxLVr1VkXxQJ8Ccwz4GlinOzUAeFtK2byk14UpLQID2kSZ0dOnc/vPHdgPHULl9957YqLMTE0mY3aNITwhnN+6/0Yt+1pFMEADR76BvR+DY20Y9Au41C/8eIoST9ztNLboggNCdcEBrWo70dtXBQeUdPQpMDXRJqxsoTt1DHgL7R4WfyllwbeXFxNKk8CALlHm11+T8H/LsG7XFvd58zCxzjtRZlxqHAO3DKSiRUV+7f4rVuZWRTPi0iFYNxrupUDP+eBdsE2hitJJRHwKm4Ki2BgcnR0c0MmzEn1UcMBT50ZKOmFRSdR2scG9YuH+3lUUWT4obQJzn5urf+f6xx9j8UxdPBYvwbySS57t/475m1d3vUqPmj34tPWnRV+cvX1dKzJXjmh3/3f5HMyVa0ShDQ4IvHqLTcFRbAmJ5mZqBhWtzOnurc0c0KiqCg7QF1JKYpO1YhIWnURYVDJhUUlc1+1jmtnTk5daFa5Muz5nMO5oC4610p06BLz5pFT4JYHSKjAAKQcPEjXpLUzs7PBYsgTLZ+rm2X5xyGIWBS/iw5Yf0q9Ovzzb5ousTNj7ERyZD1V8tS6zivkLpVaUDe4HB2wIimZ3juCAPr5u9PZ1o7aLWsfLL1JKIm/e5cx9IYlOIiwqiRsp9wBt3E0tZxsautrS0M2OBq52NHSzLbSbUp8CsxttBNkK3alhwItSyk6FsqwYUZoFBiDt7FmujR2HJjUV9wXzsW75+MJjWZosxv81nsC4QFY9v4pnHJ7RjxH/bIMN47Wf8L5L4Zmu+hlXUapISc9kZ9h1Ngb/Fxzg5WZHb19XFRzwEBqN5EpiavbM5IxOUG6lalNAmZoI6rjY0NDNLltQ6lexxdqicMULc0OfAvPIQn5JX9y/T2kXGICM69e5NnYc6RERVPlwFvb9+z+2bcLdBAZtGUR58/Ks7r4am3J6+gWZeElbxOx6KLR+Czq8r83WrFDkQlxyGltCY9gYFMXpqP+CA/r4utGlYWVs9PhFWdzJ0kguxqc84OIKj07mdnomAOVMTXimcgUautnqZiV21KtcAUtzU4PapU+B2QP8xH+5v4aiLebVschWGpmyIDAAWSkpRL05iTtHjuA4fhzOb7zxWD93QGwAL+98mY5VOzK33Vz9+cMz0mDHNAhYDtXbaPOaVchnlU1FmeVCXAqbgqPYGBzFtcS7WJqb0MmzMn18XUtdZc6MLA3nY1N0s5IkwqKTCY9O5m6Gts6ihZkJnq62NNS5txq42lG3UgWjBEjoU2CqoV2DaYF2R/9R4HUp5bU8O5YAyorAAMiMDK5/9BG31q7DtmdPqnz6CSblcq/R/mPYj3wd8DXvNnuXofWG6teQ4N+0aWYsbWHAT1C91ZP7KMo82uCAm2wMimZrqDY4wMG6HN29qtDHz7XEBQekZ2bx7/Xb2eslZ6KSOHv9dnYtHutypjRwtaOB231BsaOWszVmxURQDV3RcpKU8ptCWVaMKEsCA9o/0oT/W0b8vHlYNW6M+7cLMbW3f6SdRmp4Y+8bHIk+wopuK2jo1FC/hsSe0brMEi9BxxnQ8g2VMFORbzKyNBw8F8/G4Gh2nblOeqaGqg5W9PZ1LZbBAXfvZREek6xbgNe6us7F3iZTo/3utbU0066XuNnRQLdmUsPRGhOT4iuYRi+ZXBIoawJzn6Rt24iZ/g7m7u7aRJkeHo+2SU9i4JaBmAgTfu/xO3YWdvo1Ii0ZNr8O4RvhmeehzyIon/+s0AoFPD44oI+fGz19quBS4ekGB6SkZ2a7t87oFuEvxKWg0xIcrMs9sPje0NUOD4fyJWr2BYYXmGtSyke/lUoYZVVgAFJPnSLytYlgaorH4kWU93k0409ofCgjd4yktVtrFnRYoP8/Ainh+FLY9T7YumpDmV1LfOyIwkjEJaexOSSaTcHRTyU4ICk1Izsc+L6gXLxxJ/u6SwULvNzsaJBDUKrYWZY4MckNNYPJB2VZYADSL17i2tixZMbF4Tr3S2w7PRp5vursKmafmM3/Gv+PkQ1G5jKKHrh2EtaOgjvx0O0L8B+lEmYqisT94IANQVFE3vwvOKCvnytt6hQ8OOD+7vcz0cnZ4cHXEu9mX3ezL0/DHOslDVxtS3VodZEF5jFp+kGbUbm8lLLExwqWdYEByExMJHL8BO6GhuIybSoOI0c+8AtLSsnkA5PZe3UvP3X9CT+X/JUEKDB3EuCPMRCxF7yHQI95UC7vNDcKxZO4HxywISiKbaEx2cEBPbyr0NvXjUZV7R/5vMfdTud05H+7389EJxGT9F8Vz2qOVtnurfvRXA7WuQfMlFZUqph8oARGiyYtjeip07i9axcVhw2j0jvTEab/xdHfvnebwVsHE5MSQ33H+vg4++Dn4oeviy8uVnmnoSmYIVlwcC7s/xyc68HgFeBUR3/jK8o09zLvBwdEsTs8Njs4oJePK0C2oNxI0RbCFQJqOlnjlb0Ab4enqy125VWSTiUw+UAJzH9IjYa4uV+R+OOP2HTogNtXczGx+i8RXlRKFOvOrSM4LpiwG2GkZWl/0blau+LjohMcZ1/qVKyDmUkRJ7cRe2H9GMhMh14LoaEeUtcoFDm4nZbBzjOxbAyK4mjEDYQw/O730oQSmHygBOZREn/9ldhPPsWyfn08lizGzNn5kTYZmgz+TfyX4LhgguODCYoLIi41DoDyZuXxdvLG18UXXxdfvJ29sS1nW3BDkqK06zKRJ6DZOOj0MZiVLTeE4ulwK/UeluamBt/9XppQApMPlMDkzu19+4h6ezJmFSvisXQJFnXydlNJKbl+53q22ATHBXPu5jmyZBYCQS37WlrBcdaKTtUKVfMXSZOVAbtnwt/fgVtjGLgc7Et88KJCUeJRApMPlMA8nrtnzhA5bjyatDTcFy7AunnzAvVPzUgl7EaYVnDigwmJD+H2vdsAOFg64OPsg6+LL34ufng6emJhavH4wcI3wcbXwNQc+v0f1HmuKI+mUCiKiBKYfKAEJm8yoqO1iTIvX6bKxx9h36dPocfSSA0Xb10kOD4427V2JfkKAGYmZng6euLr7JsdPOBU3unBAW5c0O7+jwuHdlOh3TQwUS4NhcIYKIHJB0pgnkzW7dtEvfkmd44ew2nCeJwmTkToKa1LYloiIXEhBMUHERIXQtiNMO5ptPUr3GzctDMcZ63g1LavjWlmOmz/HwSvgprttQkzrZ3yvIdCodA/SmDygRKY/CEzMoiZNYuk9X9g3aoVrl/OwczBQe/3ycjK4GziWYLiggiJDyEoLogbd28AYG1ujZeTl3aGk3QDr8OLqGBZUbsuU7WZ3m1RKBSPRwlMPlACk3+klNxau5bYTz7FtGJF3OZ9hZW/v8HvGZUSle1WC4kP4dzNc2ikBoGgTpbEN/UOvnV64ttiMu4VPEpFGg6ForijBCYfKIEpOGn//EPUm5O4FxmJ86Q3cXz5Zb25zPJDyr0UTt84rRWdmJOExgaQgjbFuaOlA766/Ti+Lr54OnpSzlSFNisU+qZYCIwQoiswHzAFlkkpZz903QL4BfAHEoDBUsrLumvewFLAFtAATQBz4FCOIdyBlVLKSUKIqsDPgL3uftOllNvzsk8JTOHISknh+owZJG//E5t27agy+3PMKhonE3JWViYRBz8lOOj/CK7gQLCdC9fuxgJgbmJOA8cG2XtyfJx9Hg0eUCgUBcboAiOEMAXOAZ2ASOAkMFRKGZ6jzQTAW0o5TggxBOgrpRwshDADAoHhUsoQIYQjcEtKmfXQPQKAt6SUB4UQ3wNBUsrFQghPYLuUsnpeNiqBKTxSSm7+9htxn8/G1MlJ6zLzM1Cesvxw5Zh2Y2baLW50/piQSrWyXWtnEs6QodHWK/eo4JE9w/F18aWWXS1MVTSaQlEg8iswhsyD0BS4IKW8qDNoNdAbCM/RpjcwS/d6HfCt0DrROwOhUsoQACllwsODCyHqAi78N6ORaGc7AHZAtD4fRvEgQggcXniB8t4+RL31FleGj8Bl8mQcRo00zjpItRYw7hCsfxmn7VPo6DeMjs/PBfPypGelczbhLMFx2o2gR6KPsOXiFgBszG3wcfbBx8UHX2dt5gFrc5VkU6HQB4acwQwAukopx+iOhwPNpJQTc7QJ07WJ1B1HAM2AYWjdZi6AM7BaSjnnofFnALZSyv/pjqsAu4CKgDXwnJQyIBe7XgVeBahatar/lStX9PrcZZGs5GRi3nuP27v/wqZjR1w/+xRTOz0XKMsvmixtssyDX0IlLxj0MzjWeqCJlJLI25H/ZR6ID+bCzQtIJCbChLoV62ZvBG3p2hIHS/1HzCkUJZni4CIrisCMAl5Du+6SCuwB3pdS7snRNxytCy1Ad/y27nm+EkK0AH4AGkopNY+zUbnI9IeUkpsrVhD75VzMXVxw++Zrynt5Gc+gc7vgj1dAarTVMuv3zLP57Xu3CY0PzXarhcaHkpqZSkWLiix4dgG+LqoQmkJxn/wKjCHDf6KAnImj3HXncm2jW3exQ7vYHwkclFLekFKmAtuBRvc7CSF8ALOHZigvA2sApJTHAEtAreg+JYQQOIwYQfVVK0FKLr/wIokrVmK0KMW6nbUuM8fa8Psw2PmeNrfZY6hQrgKt3Frxmu9r/F/n/+PI0COs6LYCm3I2jNk1ht1Xdj9F4xWK0oEhBeYkUEcIUUMIUQ4YAmx+qM1m4H6ZxAHAXqn9RtoJeAkhrHTC044H126GAr89NNZVoCOAEKI+WoGJ1+PzKPJBeW9vavyxHpvWrYn99FOiJjkaCJ0AACAASURBVL1F1u3bxjHGviqM3gFNXoFj38LyHpCcv6U5MxMzfF3+v707j4+quv8//vpkJhsBQlb2hARFtgTLpogsCnXB2E2/glVrH7XSVtxXqNWKrRsuqF/xV7W2arVqv7ZaCSgWF0CBAiJLiKwJWyBk38g+c35/3JtkCNsFM5ksn+fjMY/MzF3mnITkzTn33HPO5s1pb3JW9Fnc9cVdvL7l9cAFplLtkN8CxhhTD9yMFRbfAv8wxmwRkYdF5Af2bq8CMSKyE7gTmG0fWww8gxVSG4D1xphFPqe/iqMD5i7gRhHZaG/7udG/BgHh6tGDfi8uIP6eeyhfupTsK66kOjPz5Af6gzsULnvKmlYmdzP8aQJkfeH48OiwaF696FWmJk7lqXVP8diax/B4PSc/UCmlN1rqNRj/qly/npw77sRTXEzP386hx/TpgbvbPn8bvHsdFGyHC+6HCXeBw5tEvcbL0+ue5o3MN5jcfzJPTHiCLsFdTn6gUh1QW7gGoxRdRo4k6YP36XLOOeQ+NJcDd92Np+JwYAoTdxbc+Bmk/A98/kf4+1VQWeTo0CAJ4p4x9zBn7ByW71/OL5b8onGeNKXUsWnAKL9rWLgs7o47KPv4Y3ZfeSXV27YFpjChXeEnL8Nlz0D2MnhpIux33or96ZCf8twFz5FVmsU1i64hqyTLj4VVqn3TgFGtQoKCiP3VTBJffw3v4cPsvmo6Je+9F5iL5iIw5gb4xRJA4C+XwH9fBodlmdx/Mn+9+K/UeGq49qNrWZu71r/lVaqd0oBRrarLmDFWl9mokRz83QMcnD0bb2VlYArTdyT8ahkMvBA+ugf+eQPUOBvxNix2GG9d9hZx4XHM/M9MFu5a6OfCKtX+aMCoVueOiaH/K68Qe8vNlH64kOyrrqJm587AFKZLNFz9Dkx5ELa8D69cCHnfOjq0b9e+vHHpG5wddza//fK3vLTxJR3GrJQPDRgVEOJyETdrFgl/eRVPcQnZ/3MVJR98EJjCBAVZI8p+9m+oKrFCZtM/HB0aGRrJS99/ibTkNF7Y8AIPrXqocWJNpTo7DRgVUBHjxpH0/r8IT0nh4Ow5HLj/frxVVYEpTNJE6+7/Pt+zpplJvwPqqk96WIgrhEfPf5SZqTP5145/MWvpLCpqK1qhwEq1bXofjN4H0yaY+nryFyyg8E8vEXrGGfR97jlCk5MCUxhPPXz2MHz1HETEw4DxkHCeNWNz/LAT3jvz/o73eXjVwyT1SOLFKS/SK6JXKxZcqdYR8Mku2wMNmLanYsWXHLj3Xrw1NfSeO5fIy9MCV5gdS2Hj27B3FZTZ0+iFRUL/c62wSTjPau24j1w1c2XOSu5cdicR7ggWTF3A4OjBASi8Uv6jAeOABkzbVJebS85dd1P19df0mD6dnr+dQ1BoaOAKZAyU7LWCZs9K62vBdmubOwz6jrYDZxz0Hwuh3dhWtI2bPr2JitoKnpn8DOP7jg9c+ZVqYRowDmjAtF2mro7855+n8JU/EzpkCP2enU9IYmKgi9WkIt8KmobQyd1kLQ0gLuiVAonjOdR7KLOy32Nn2W4eOPcBrhh0RaBLrVSL0IBxQAOm7Sv/4gsO3jcbU19P70ceofslFwe6SMdWUw771tiBswpy1kF9NRUi3N0vka/cXm7seR43j51NUNQA62ZPpdopDRgHNGDah7oDB8i5406qNm4k6ppriL/vXoJCQk5+YCDV18CBDbB3JXW7v+KRss38MyKESysO88eaMEISxlndaonjIfYsx5NuKtUWaMA4oAHTfpjaWvKemU/Ra68RNnw4fZ+dT0i/foEulmPGU8+r/32c53a8y6igrjyXV0Rkea61MTzKun6TMA4Sz4PeI8AVHNgCK3UCGjAOaMC0P+VLl3Jgzm8B6PPYo3SbOjXAJTo1i7IW8cBXD9C3a19eHD2H/gXZsHel1a1WtMvaKbgL9BvdNDS63xgIiQhswZXyoQHjgAZM+1S7fz85t99BdUYG0ddfT/xddyJtvcvMx7rcddz2+W24g9y8cOELpMSlWBvKDzWFzd6VkJsBGAhyW62ahhZOwjhrihulAkQDxgENmPbLW1tL3rwnKX7zTcJGpNJv/nyC+/QJdLEcyyrN4qalN1FYVcgTE5/gwoQLj96putQaONAwNDrna/DUWtvihjTdi5M4DiLbT3ehav80YBzQgGn/yj7+mIP3/w5xu+n9xON0mzw50EVyrKCqgFs/u5WMggzuG3sf1wy55sQH1FXDgfVNgbP3v1Brz/4cmdB0L07ieRA7SEeqKb/RgHFAA6ZjqN2zh/2330HNt98S88sbiLvtNiS4fVwkr6qv4r7l9/H5vs+5dsi13D36blxBLmcHez2Qu/nIG0AP51vbusRCwrlNXWq9UsHl9l9F2gJjoL7aGjJeUw41ZVBT0ex1+ZGPWnt7dDIkT4IBEyAiNtA1afM0YBzQgOk4vDU1HHr0MUrefZfwUaPo+8zTBPfsGehiOeLxenhy3ZO89e1bTEmYwmMTHiPcHX7qJzIGCncdeR2neLe1LaSrNVigIXD6jYbg0/gMf/B6jv6Df6wwONGj1v7qrT/55wW5IbSb/ehufR/ytja1BnsOtyY+TZpkfb/Cuvu3/u2QBowDGjAdT2n6InIffBAJDaXPvHl0nXB+oIvk2JuZbzJv7TxSYlN4/sLniQmP+e4nLTvQ1LrZswryMrEGDgRb86glnmc9+p8D4T2cn9cY616f47YMjhcGzVsV5VB32NlnBkfYodD1yIBofN7NCtJjve/7cIcd3X3oqYcD31jLaGcvh33/tVpD4rK+T0kTrRZO/3PaTjAHkAaMAxowHVNNVjY5t99OzY4dxPxqJnE334y420f30Kd7PmX2itnEhsfy4tQXSYps4Rmlq4qtazcNrZwD34C3DhDoOcxq3UTEWUFQW3GcgDiF1oK4TvAHv+vxgyCk+euurdvFV1cN+9dYYZO93Bpg4a0HV4gVMkkTrUffUZ3yniUNGAc0YDoub1UVuY88Qul7/6TLmDH0efopguPjA10sRzblb+KWz27BYzw8d8FzjOo5yn8fVltp/fFsuI6zb43VogjucmqtgiMe3U/cWmiPasph72rI+sIKnNzNgLFaVYnnNQVOr9ROMSuDBowDGjAdX8kHH5A792GCunSh71NPEjFuXKCL5Mi+8n3ctPQmcipyeOT8R7g06dLW+WCvx+r+6ugDAr6ryiLY/WVTl1rD7NrhUTDgfOv6TdLEDjuaTwPGAQ2YzqFm507233Y7tVlZxM6aRexvfo24HI7UCqDSmlJu/exW1uet57aRt3HD8BuQDvjHqkMoOwi7V0DWMit0SvdZ73ft1dS6SZoIUW1oRvDvoE0EjIhcAjwHuIA/G2Meb7Y9FHgDGAUUAtONMbvtbanAS0B3wAuMAYKBFT6n6Ae8aYy5XUTmAxfY73cB4o0xJ7xqqQHTeXgrK8mdO5fSf39IxHnj6DNvHu7Ytj8ctcZTwwNfPsBHuz/iykFXcv859+MO0tZFm2aMNXove3lTC6dh+HjUgKYRagMmQLf2MdKxuYAHjIi4gO3A94H9wFrgamNMps8+NwGpxphfi8gM4MfGmOki4gbWA9cZYzaKSAxQYozxNPuMr4E7jDHLm71/C/A9Y8wvTlRGDZjOxRhD6T//Se4f/oire3f6PP0UEWPHBrpYJ+U1Xp5f/zyvZrzK+L7jeXrS00QE69xk7YYxkL+1acDA7hXWLA1gzcjQ0LoZMN7qYmsH2kLAjAMeMsZcbL+eA2CMecxnnyX2PqvsUMkF4oBLgZ8aY649wfkHAZ8CCaZZJURkJfB7Y8x/TlRGDZjOqXrbNnJuu53avXuJu/VWYmbeiLSDC7P/t/3/eGT1I5wZdSYLpiwgvkv7GLSgmvF64ODGpsDZuwrqKgGx5pxLtq/fJIxrs5OctoWAuRK4xBjzS/v1dcA5xpibffbJsPfZb7/eBZwDXIvVbRaPFTjvGGPmNTv/g0B3Y8zdzd5PBFYD/Zq3eJrTgOm8PBWHyX3wQcoWLyZiwgT6zHsCd1Tb/9/jiv0ruHvZ3XQL6caLU19kUNSgQBdJfVf1tdYCddnLrWs4+9daQ8eDgq0bYhsGDPQbDe4ALh3uw2nAtNX/trmB84Fr7K8/FpEpzfaZAbx9jGNnAO8dL1xEZKaIrBORdfn5+S1ZZtWOuLpG0Ofpp+j10O+pXL2a7B/9mMr16wNdrJOa0G8Cr1/6OsYYfvbRz1h5YGWgi6S+K3eINdR58mz4xUcwew9c+y8YN8u6mXX5PHhtGjyeCG/8CL6cb9+Xc8L/P7cJ/gyYHKC/z+t+9nvH3MfuIovEuti/H1hujCkwxlQCi4GRDQeJyAjAbYz5+hife7zgAcAY87IxZrQxZnRcXNyp10p1GCJC1IwZDHj3HSQ0lD3X/YzCV1/FeL2BLtoJDY4ezFuXvUXviN7MWjqL93e8H+giqZYUEgFnTIHvz4WZn8O92TDj7zDqeqg4BEsfglcuhCeS4O2rYfWf4FCmda2njfFnF5kb6yL/FKwgWYt1XWWLzz6zgBSfi/w/McZcJSJRWNdXzgdqgY+B+caYRfZxjwM1xpjfN/vMwfa+Sc2vyxyLdpGpBp7ycg7+7gHKlyyh6+TJ9Hn8MVw9TmHqlAAory3nzi/uZPXB1fwq9VfMOnuWDmPuDCrymq7fZC+H4mzr/Yg4a2Raw7Q2UUl+uwcn4Ndg7EJMA57FGqb8F2PMIyLyMLDOGPOhiIQBfwO+BxQBM4wxWfax1wJzAAMsNsbc63PeLGCaMWZrs897CAgzxsx2Uj4NGOXLGEPxm29xaN483HGx9Js/n/ARIwJdrBOq89bx8KqH+WDnB1yefDlzz5tLcCecuqRTK9nbFDZZy6DCXoo7sv+R9+B0b7n1ktpEwLR1GjDqWKo2bybn9juoy8ujx09+QnDvXrh6ROGKjsIdHY0rKgpXdDSuyMg2MfrMGMPLm17mhQ0vMLbXWOZfMJ/uIToDcKdkDBTubJrSZvcKa/45gJgzjwyc77AqqgaMAxow6ng8paXkzp1LxbLleA8fZ7bfoCBckZG4oqNxR0U1BU90lP3a53l0NK7oaIL8uLTzwl0LeXDlgyR2S2TB1AX07drXb5+l2gmvFw5tbmrh7FlpTWIKMO0pGHvjaZ1WA8YBDRjlhLe2Fk9xMZ6iIjzFxdQX2c9LiqkvKsJjv64vKbael5RYv9jHENSlS2PYuKJ64I6ynruj7YCK8nkeHU1Q166ndF1lzcE13P757YS4QlgwZQHDYoe11LehwzDGkFORw5bCLWwp2MLWoq0kdk8kbWAaqbGpHfs6lqfOmkE7axmcdQn0Sjmt02jAOKABo/zBeL14SksbQ6m+2A6e4mYBVVxsbyvC1NQc+2TBwbh79Dhmy8gV1dBlZ4dVdDSuHj3IqtjDTUtvorimmCcnPsmk/pNa9xvQxuRX5pNRkMGWwi1kFGaQWZBJcY3VbRQcFExyZDK7y3ZT46khoVsCaclpXJZ8GQndEwJc8rZLA8YBDRjVFhhjMFVVVvA0hpDdMioupr64qZXUEEresrLjni8oMhLp0Z1sKeRQSBXJiWczKGmMHUh2d12PpudB4R1nAa3SmlK2FGyxwqQgg4zCDPIq8wAIkiAG9hjI8JjhDI8dzrDYYQzqMYhgVzDlteUs3bOU9Kx01uauxWAYETeCtOQ0Lh5wMVFhbf8m3NakAeOABoxqr0xdHZ6SkhOGUm1hATn7t+IqrSCyWgjyHPt3XcLC7NaRNYAhZMAAwlNTCE9NJTgxsc12GVXWVZJZmNnY1ZVRmMG+8n2N2wd0H8DQmKEMj7UCZXD0YEdLUecezmVx9mIW7lrIzpKduMXN+f3OJy05jUn9JhHmDvNntdoFDRgHNGBUR+fxenh8zeO8s/VtLou/gPvPuhlXaSWeErv77oiWURGewiJqsrMxlZWA1RoKT0khPDWFsNRUwlNTcUef/uij01XrqWVb0TYyCjPIKMggszCTrNIsvMa61tU7ojfDY4c3BsrQmKHfeSSdMYbtxdtJz0pncdZi8qry6BrclYsGXERachqjeo4iSAI/ijAQNGAc0IBRnYExhjcy3+CpdU9xdtzZPH/h8yfs8jEeDzU7d1G1aSPVmzZTtXkzNdu3Nw5cCO7bl/ARqYSlpFpfhwxp0W62em89u0p2NXVzFWSwo2QH9fYSzdFh0VarJMbq5hoaM5TYcP8uveDxeliTu4b0rHSW7llKZX0lvSJ6MS1pGpcnX84ZUWf49fPbGg0YBzRgVGfyye5PmLNiDr0ievHi1BdJ7O588StvZSXVmZlUbdxE1ebNVG3aSP2Bg9ZGl4vQQYMIT01t7FoLSU52tKib13jZW7aXjMIMq5urIIOtRVup9lQD0C24G0NjhzaGyfCY4fSK6BXQbrvKukq+2PcF6VnprDywEo/xMDh6MGnJaUxLmkZcl44/BZUGjAMaMKqz2ZC3gVs+uwWA/73wfzk7/uzTPld9fr4dNpsaWzre8nIAgiIiCBs+3OpaS0khfMQI3PHxHDx8sLFlsqVgC5mFmZTXWceEucIYEjOEYTHDrIvwMcNI6J7QpruhCqoKWLJ7Cem70skozCBIgjin1zmkDUxjasJUugR3CXQR/UIDxgENGNUZ7S3by2+W/obcw7k8OuFRLh5wcYuc13i91O7e09i1Vr5hPXXbdyD11qy/Jd2C2NbbsLOPkN3XhWvIIM7sO6JxRFdyZHK7Xq0zqzSLRVmLWJS1iJyKHMLd4VzQ/wIuH3g55/Y+t13XrTkNGAc0YFRnVVxdzK2f3cqG/A3cNeourh92/XfqdiqrLWscHtwwoiv3cC7B9YakPOGc4miGHwqlz97DhB4ssg4SIfSMgda1HLtrLfTMM5Hg9j2XmjGGb/K+IT0rnSW7l1BWW0Z0WDTTkqaRNjCNodFD2+zIPKc0YBzQgFGdWXV9Nfd/eT+f7PmE6WdNZ/bY2Y7+l11ZV8nWoq1NXV2FW9hTtqdxe0K3BIbFDLOumcQOZ0j0kCO6iuqLi6nOyGjqWtu0CU+xdeOjhIURNnSoNXJtRCphqakE9+3bbv8g13pqWZGzgvRd6Szbv4w6bx1JkUmNN3P6czofb00N9fn51OflW1/z86nPy2t8Hn3dtXSddHo34WrAOKABozo7r/Hy7NfP8tctf2VSv0nMmzjviDCo89SxvXj7EXfC7yrZ1Tg8uGeXno3XS4bFDmNYzDAiQyNPqQzGGOpycqjauLExcKozMxtnN3BFRxOekkJYagrhqSMITxne5pdSOJbSmlL+s+c/LNy1kPV51uJ2I+NHkjYwjYsSL3L8ffMePtwUGPajzic4rEcB3tLSow92uXDHxuKOiyNm5o10v+ii06qLBowDGjBKWd7Z+g6PrXmMs6LOYsbgGdYNjAVb2Fa8jTpvHQBRoVGNIdJw86K/hgebujpqduygatMmqjZZo9Zqd2U1LqoVnJhgh43V0gkdPJig0LaxnLATORU5LM5azMKshWSXZhMsbr4fM55p3ccxIigBKSqxWh5HBUf+MSdfleBg3HFxuOPjra9xcbjj4456zxUV1SIzgGvAOKABo1STZfuWcc/ye6iqryIiOOKIVsnw2OH0iegT0K4qT0WF3bW2merNm6jauIn6PGsaGIKDCRs8+IiWTsiAxIAvp2CMsWZcyDsyJHyDozI3B09BAe7ao5dAlvDwowPD5xFsh0dQZGSr/mw0YBzQgFHqSAVVBZTVljGg+4A2PTy4Qd2hQ1bX2ubNVG3cRHVGBt6GWQi6dyd8+PCmrrXUFNyxLdPiMh6PNRPCSbqqPPkFmLq6o44P6trVJzjskIiNITu4hJU1mXxWuZHc8Gqio/ty2cA00pLTSIpMapGytwQNGAc0YJTqWIzHQ82uXVbg2NdzarZvB4/VOnD36X1E11rY0KEEdWm65mTq66kvLGxqYeQd3eKoz8+nvrCw8Zy+XJGRPi2N+KNbHfHxuGNjj/jMY6msq+TTvZ+SnpXO6oOr8Rovw2KGcfnAy7lkwCXEhMe07DfuFGnAOKABo1TH562qsmYhsK/lVG/aTF1OjrXR5SJ04EAICrJaHEVFjdd5GolYa/acpKvKFRfnlwXl8ivzWZy9mEVZi/i26Ftc4mJcn3Fcnnw5FyRc4GgCz5amAeOABoxSnVN9YaE1Wm3zZqoyMpAg15EXxH0vkEdHt5l7c3YW7yQ9K51F2YvIPZxLF3cXpiZOJS05jbG9xuIKOvn0PC1BA8YBDRilVHvkNV6+PvQ1i7IW8cnuTyivKycuPK7xZs6zos7y60V/DRgHNGCUUu1djaeGZfuWkZ6VzoqcFdR76zmjxxmNN3P2iujV4p+pAeOABoxSqiMpqS6xJt/MSmdD/gYEYUyvMaQlpzE1cSrdQrq1yOdowDigAaOU6qj2le0jPTudRVmL2FO2h1BXKJP7TyYtOY3xfcYT7Dr960oaMA5owCilOjpjDBkFGSzMWsjH2R9TXFNMj9AezBk7h2nJ007rnE4DpuPMH62UUuooIkJKXAopcSncM+YeVh1YRfqudHp37e33z9aAUUqpTiI4KJiJ/SYysd/EVvk8v84FISKXiMg2EdkpIrOPsT1URN61t/9XRAb4bEsVkVUiskVENotImIh0E5ENPo8CEXnW55irRCTTPubv/qybUkqpE/NbC0ZEXMAC4PvAfmCtiHxojMn02e0GoNgYc4aIzACeAKaLiBt4E7jOGLNRRGKAOmNMNXC2z2d8DfzLfn4mMAcYb4wpFpF4f9VNKaXUyfmzBTMW2GmMyTLG1ALvAD9sts8Pgdft5+8BU8S6O+giYJMxZiOAMabQGHPExD8iMgiIB1bYb90ILDDGFNvH5PmhTkoppRzyZ8D0Bfb5vN5vv3fMfYwx9UApEAMMAoyILBGR9SJy7zHOPwN41zQNgxsEDBKRr0RktYhc0oJ1UUopdYra6kV+N3A+MAaoBD61h8V96rPPDOC6ZsecCUwG+gHLRSTFGFPie2IRmQnMBEhISPBbBZRSqrPzZwsmB+jv87qf/d4x97Gvu0QChVitneXGmAJjTCWwGBjZcJCIjADcxpivfc61H/jQGFNnjMkGtmMFzhGMMS8bY0YbY0bHxcV91zoqpZQ6Dn8GzFrgTBFJEpEQrBbHh832+RC43n5+JfCZ3eW1BEgRkS528EwCfAcHXA283excH2C1XhCRWKwus6yWq45SSqlT4bcuMmNMvYjcjBUWLuAvxpgtIvIwsM4Y8yHwKvA3EdkJFGGFEPYosGewQsoAi40xi3xOfxXQ/BbUJcBFIpIJeIB7jDGF/qqfUkqpE+vUU8WISD6w5zQPjwUKWrA47YHWuXPQOncO36XOicaYk15j6NQB812IyDonc/F0JFrnzkHr3Dm0Rp39eie/UkqpzksDRimllF9owJy+lwNdgADQOncOWufOwe911mswSiml/EJbMEoppfxCA+YkHCw58Gt7OYENIvKliAwNRDlb0snq7LPfFSJiRKTdj75x8HP+uYjk+ywV8ctAlLMlOfk5d6QlMBz8jOf7/Hy3i0jJsc7Tnjioc4KIfC4i34jIJhE5vSUuj8cYo4/jPLBuEN0FJAMhwEZgaLN9uvs8/wHwcaDL7e862/t1A5YDq4HRgS53K/ycfw68EOiytnKdzwS+AaLs1/GBLrc/69ts/1uwbg4PeNn9/DN+GfiN/XwosLsly6AtmBM76ZIDxpgyn5cRWDMPtGdOllkA+APW+j3VrVk4P3Fa547ESZ070hIYp/ozPtZ0VO2NkzoboLv9PBI40JIF0IA5MSdLDiAis0RkFzAPuLWVyuYvJ62ziIwE+psjp+9pzxz9nIEr7G6E90Sk/zG2tydO6tyRlsBw+jNGRBKBJOCzViiXPzmp80PAtSKyH2tS4VtasgAaMC3AGLPAGDMQuA/4XaDL408iEgQ8A9wV6LK0soXAAGNMKvAfmhbK68h8l8C4GnhFRHoEtEStYwbwnmm2yGEHdTXwmjGmH9b8jn+zf8dbhAbMiTlZcsDXO8CP/Foi/ztZnbsBw4EvRGQ3cC7wYTu/0H/Sn7OxVlWtsV/+GRjVSmXzFyf/th0tgdFOnMrv8gzaf/cYOKvzDcA/AIwxq4AwrDnKWoQGzImddMkBEfH9hbsM2NGK5fOHE9bZGFNqjIk1xgwwxgzAusj/A2PMusAUt0U4+Tn39nn5A+DbViyfPzhZTqMjLYHhpL6IyGAgCljVyuXzByd13gtMARCRIVgBk99SBWirK1q2CcbZkgM3i8hUoA4opml9m3bJYZ07FId1vlVEfgDUYy0t8fOAFbgFOKxzh1kC4xT+Xc8A3jH2sKr2zGGd78Lq+rwD64L/z1uy7nonv1JKKb/QLjKllFJ+oQGjlFLKLzRglFJK+YUGjFJKKb/QgFFKKeUXGjBK+YmIPCQid7eBcuy272NRqlVpwCillPILDRilToGIRIjIIhHZKCIZIjLdt4UgIqNF5AufQ0aIyCoR2SEiN9r79BaR5fa6IxkiMsF+//+JyDp77ZW5Pp+5W0Qes/dfJyIjRWSJiOwSkV/b+0y2z7nIXv/jT8eaU0pErhWRNfa5XhIRlz+/X6pz04BR6tRcAhwwxowwxgwHPj7J/qnAhcA44EER6QP8FFhijDkbGAFssPe93xgz2j5mkoik+pxnr73/CuA14EqseeDm+uwzFms23KHAQOAnvgWxpwKZDoy3z+UBrjmFuit1SnSqGKVOzWbgaRF5Akg3xqwQkRPt/29jTBVQJSKfY4XAWuAvIhIMfGCMaQiYq0RkJtbvZW+soNhkb2uYymQz0NUYUw6Ui0iNzwzHa4wxWQAi8jZwPvCeT1mmYE3SudYuczjQntd4UW2cBoxSp8AYs91eD2ca8EcR+RRrfrKG3oCw5occfQqzXEQmYk2O+pqIPIPVMrkbGGOMKRaRZHPFigAAASNJREFU15qdq2EmZ6/P84bXDb/HR31Ws9cCvG6MmXOSairVIrSLTKlTYHdxVRpj3gSeBEYCu2mavv+KZof8UETCRCQGa2bitfaCVoeMMa9gTf0/EmtVwcNAqYj0BC49jeKNtWfODcLqCvuy2fZPgStFJN6uS7RdFqX8QlswSp2aFOBJEfFizaD9G6yupldF5A/AF8323wR8jrXGxh+MMQdE5HrgHhGpAyqAnxljskXkG2Ar1iqEX51G2dYCLwBn2J/5vu9GY0ymiPwO+MQOoTpgFrDnND5LqZPS2ZSV6gBEZDJwtzEmLdBlUaqBdpEppZTyC23BKKWU8gttwSillPILDRillFJ+oQGjlFLKLzRglFJK+YUGjFJKKb/QgFFKKeUX/x/vTr6qO4L82AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3.best_score_, gsearch3.best_params_))\n",
    "test_means = gsearch3.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/vincent/.local/lib/python3.6/site-packages/sklearn/preprocessing/label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'reg_alpha': 1.5, 'reg_lambda': 1}, -0.06803248813467347)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#找出最佳 alpha & lambda\n",
    "\n",
    "reg_alpha = [ 1.5, 2]    #default = 0, 测试0.1,1，1.5，2\n",
    "reg_lambda = [0.5, 1, 2]      #default = 1，测试0.1， 0.5， 1，2\n",
    "\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "\n",
    "\n",
    "xgb5 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.8,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "gsearch5_1 = GridSearchCV(xgb5, param_grid = param_test5_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch5_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch5_1.best_params_,     gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#以上为参数调试过程，可以根据实际情况通过调整补长以及区间来不断的优化性能， 因为时间关系就不再进一步处理。\n",
    "\n",
    "最终模型参数如下：\n",
    "#reg_alpha': 1.5, 'reg_lambda': 1\n",
    "#colsample_bytree': 0.8, 'subsample': 0.8\n",
    "#learning_rate =0.1\n",
    "#n_estimators=98\n",
    "#max_depth=3\n",
    "#min_child_weight=1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=0.8,\n",
       "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=1, missing=None, n_estimators=98,\n",
       "       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n",
       "       reg_alpha=1.5, reg_lambda=0.8, scale_pos_weight=1, seed=3,\n",
       "       silent=True, subsample=0.3)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb5 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.8,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=1.5,\n",
    "        reg_lambda=0.8,\n",
    "        seed=3)\n",
    "\n",
    "xgb5.fit(X_train,y_train,eval_metric='auc')\n",
    "\n",
    "#设置验证集合 verbose=False不打印过程\n",
    "xgb5.fit(X_train, y_train,eval_set=[(X_train, y_train)],eval_metric='auc',verbose=False)\n",
    "\n",
    "#获取验证集合结果\n",
    "#evals_result = xgb5.evals_result()\n",
    "#y_true, y_pred = y_test, xgb5.predict(X_test)\n",
    "#print\"Accuracy : %.4g\" % metrics.accuracy_score(y_true, y_pred)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#最终模型的特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Monthly_Income</td>\n",
       "      <td>0.277542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Existing_EMI</td>\n",
       "      <td>0.122881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Interest_Rate</td>\n",
       "      <td>0.101695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Var5</td>\n",
       "      <td>0.091102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>EMI_Loan_Submitted</td>\n",
       "      <td>0.086864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Processing_Fee</td>\n",
       "      <td>0.076271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Var4</td>\n",
       "      <td>0.069915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Loan_Amount_Submitted</td>\n",
       "      <td>0.067797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Loan_Amount_Applied</td>\n",
       "      <td>0.067797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loan_Tenure_Applied</td>\n",
       "      <td>0.025424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Loan_Tenure_Submitted</td>\n",
       "      <td>0.010593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gender</td>\n",
       "      <td>0.002119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>City</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mobile_Verified</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Salary_Account</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Employer_Name</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Filled_Form</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Device_Type</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Var2</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Source</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Var1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  columns  importance\n",
       "2          Monthly_Income    0.277542\n",
       "5            Existing_EMI    0.122881\n",
       "13          Interest_Rate    0.101695\n",
       "9                    Var5    0.091102\n",
       "15     EMI_Loan_Submitted    0.086864\n",
       "14         Processing_Fee    0.076271\n",
       "20                   Var4    0.069915\n",
       "11  Loan_Amount_Submitted    0.067797\n",
       "3     Loan_Amount_Applied    0.067797\n",
       "4     Loan_Tenure_Applied    0.025424\n",
       "12  Loan_Tenure_Submitted    0.010593\n",
       "0                  Gender    0.002119\n",
       "1                    City    0.000000\n",
       "8         Mobile_Verified    0.000000\n",
       "7          Salary_Account    0.000000\n",
       "6           Employer_Name    0.000000\n",
       "16            Filled_Form    0.000000\n",
       "17            Device_Type    0.000000\n",
       "18                   Var2    0.000000\n",
       "19                 Source    0.000000\n",
       "10                   Var1    0.000000"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"columns\":list(X_train.columns), \"importance\":list(xgb5.feature_importances_)})\n",
    "df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
